Hortonworks HA Namenodes gives an error "Operation category READ is not supported in state standby"

My hadoop cluster HA active namenode (host1) suddenly switch to standby namenode(host2). I could not found any error in hadoop logs (in any server) to identify the root cause. After switching the...

Hive queries fail when the hive.execution.engine is set to MR, they work when set to Tez?

I am using HDP 2.1 sandbox for my work. The version of hive as listed by the jar file is: hive-exec-0.13.0.2.1.1.0-385.jar. I have created a directory in HDFS having weather information. the...

Oracle Virtual Box error: failure to open a session with Hortonworks

I've researched the questions already on stackoverflow that suggest upgrading to the most recent version of Virtual Box; one question at the time suggested upgrading to V4.3.14. Well, I'm on V...

Kerberized Hadoop Hive Beeline access issue

I am trying to get hiveserver2 via beeline to work with a kerberized HDP 2.3 cluster. I am on amazon ec2. Once I get a renewable ticket i am able to perform hdfs operations and also launch mr...

Hive: acquire explicit exclusive lock

Configuration (hortonworks) hive: BUILD hive-1.2.1.2.3.0.0 Hadoop 2.7.1.2.3.0.0-2557 I'm trying to execute lock table event_metadata EXCLUSIVE; Hive response: Error while processing...

ERROR 1066: Unable to open iterator for alias in Pig, Generic solution

A very common, error message in Apache Pig is: ERROR 1066: Unable to open iterator for alias There are several questions where this error is mentioned, but none of them give a generic approach...

Sqoop import : composite primary key and textual primary key

Stack : Installed HDP-2.3.2.0-2950 using Ambari 2.1 The source DB schema is on sql server and it contains several tables which either have primary key as : A varchar Composite - two varchar...

Can I install Confluent on HDP 2.4 platform

I'm trying to install Confluent over HDP for Kafka Streams which think may not be possible could you people suggest me what to do

Apache NiFi - OutOfMemory Error: GC overhead limit exceeded on SplitText processor

I am trying to use NiFi to process large CSV files (potentially billions of records each) using HDF 1.2. I've implemented my flow, and everything is working fine for small files. The problem is...

Spark num-executors

I have setup a 10 node HDP platform on AWS. Below is my configuration 2 Servers - Name Node and Standby Name node 7 Data Nodes and each node has 40 vCPU and 160 GB of memory. I am trying to...

Drop Hive external table WITHOUT removing data

The goal is to destroy a Hive schema but keep the data underneath. Given a Hive external table, created for example with script 1, it can be dropped with script 2. This deletes the data (removes...

kinit: Client's credentials have been revoked while getting initial credentials

I have hdp cluster configured with kerberos with AD. All HDP service accounts have principals and keytabs generated including spark. I know service accounts will not have passwords and set to...

Apache NIFI Install failed as Ambari service - Configuration parameter 'kafka_broker_hosts' was not found in configurations dictionary

My system environment are as follows: Using 2 node HDP 2.5 cluster with Kafka/ZK running on each. Node1: Ambari-server, Ambari-agent Node2 : Ambari-agent I have set up both as kafka brokers with...

How to configure Apache NiFi for a Kerberized Hadoop Cluster

I have Apache NiFi running standalone and its working fine. But, when I am trying to setup Apache NiFi to access Hive or HDFS Kerberized Cloudera Hadoop Cluster. I am getting issues. Can someone...

No KeyProvider is configured, cannot access an encrypted file

I have data in an encrypted zone in HDFS. I can read data with hive user, but when I create a hive table and try to query it via beeline I get this exception: Error: java.io.IOException:...

How to disable Transparent Huge Pages (THP) in Ubuntu 16.04LTS

I am setting up an ambari cluster with 3 virtualbox VMs running Ubuntu 16.04LTS. However I get the below warning: The following hosts have Transparent Huge Pages (THP) enabled. THP should be...

Tweets data in Avro format can not be loaded

I am working on HDP (Hortonworks) and trying to collect Tweets through flume and to load stored data from Hive. The problem is select * from tweetsavro limit 1; works but select * from tweetsavro...

vertext failled Error and Mapper initialized failed - Hive

I'm using Hortonworks data platform in our server with 2 nodes. I'm running query successfully in hive. Suddenly I'm facing mapping with source table to add column to my new table, By this below...

Hbase authentication wihout Kerberos or AD/LDAP

I'm actually trying to make some custom security setups in a HDP cluster (not Kerberized). The use case is hbase and kafka must implement authorization but wihthout using kerberos. Only human...

Aggregating strings with hortonworks hadoop hive

I am trying to flatten a security table to make a single row per country. I am using Hive as the execution engine currently in hortonworks if this makes a difference to the SQL required. An...

Apache Atlas quickstart - kafka error

Env: no kerberos, no ranger, no hdfs. EC2 with ssl. Getting this error after running $ATLAS_HOME/bin/quick_start.py https://$componentPrivateDNSRecord:21443 with correct user/pass Creating sample...

Sandbox IP mapping not working on HDP Sandbox

I have downloaded the latest HDP 2.6.5 from Hortonworks website. Following the instructions in the section 'MAP SANDBOX IP TO YOUR DESIRED HOSTNAME IN THE HOSTS FILE ' from the link...

Inconsistent count results from Apache HIVE

We have the latest Hortonworks's HDP, with Hive version (3.1.0) I have a problem when trying to count the number of rows, on a given condition. The count (*) returns false value when executed side...

How to migrate data from local on-premises HDFS to Azure storage

I want to move the data from my local on-premises HDFS server to my Azure HDinsight cluster. I tried distcp command but it does not understand the data lake storage path.

Session 0x0 for server null when starting Atlas

I just installed Atlas in HDP 2.6.3 and the start up of Atlas server gave below error: /var/log/atlas/application.log 2019-12-17 23:41:30,446 INFO - [main-SendThread(1:2181):] ~ Opening socket...

HBase MasterProcWALs issue

I noticed that due to some ongoing bug, the Hbase MasterProcWALs folder has filled up my Hdfs. I wanted to know if removing the files under the MasterProcWALs folder will remove any of the data in Hbase?

How to run Spark 3.0.0 on HDP (Horthonworks)?

Is there a way to run a Spark 3.0 on HDP3 (Horthonworks)? I'm aware that there is always a standalone option, but I would like to configure YARN as a scheduler.

Nifi: how to use folderFilter for fetching files from S3

I have a requirement to access AWS S3 bucket through NIFI and process files into HDFS from the specific subfolder Ex:- S3 bucket name: my_bucket. Folders under my_bucket(S3) ABC, BDE,CEF. I have...

wget + download ambari tar ball

we are trying to download the ambari version 2.6.1 but without success ( according to https://docs.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDF3/HDF-3.1.1/bk_installing-hdf-on-hdp-p... ) wget...

How to get hortonworks data platform and cloudera distribution for hadoop latest version

I'm currently working on CDH5.13 (Cloudera Distribution Hadoop), and i have a couple of questions: 1- I want to get the latest version of CDH(6.3.3). When i try to download it, i have this message...