RC version 0.9.8.15 depend on 0.9.8.14-rc #89
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
as long as not functional Python3's ThriftClient(scanner-mutator) SerializedRead and PyPy SerializedRead. these stay out of tests
It is possible now to build in one make as many hadoop FsBrokers version
as pom(and java files) targets ready.
A new Maven project structure is applied.
src/java folder structure:
mvn/
common/
src/
pom.xml.in
assembly
fsbroker/
hadoop/
vendor-version(groupings)/
src/
pom.xml.in
assembly
FsBrokerLib/ (folder content copied to each
fsbroker-DISTRO-${HDFS_VER}/src/main/java/org/hypertable/FsBroker/lib)
thriftclient/
src/
pom.xml.in
assembly
thriftclient-hadoop-tools/
src/
pom.xml.in
assembly
hadoop_distro
ThriftClient/
hypertable-examples(temporary_here)/
CmakeFiles.txt
FSBROKER:
comment, Java doesn't mean Hadoop and viceversa as Java can include
other FsBrokers
cmakefiles at hadoop_distro create a new folder & module
mvn/fsbroker-distro-version/
it is now much easier to create a new FsBroker without double-crossing
previus versions.
How to make, It is to create new folder at
mvn/common/fsbroker/hadoop/vendor-version(groupings) with it's java and
pom files in structure of apache-2.0-7
Hadoop distro groupings, as some versions are working with the same java
code and dependecies, exceptionally to the distro version which is
populated with @HDFS_VER@
these are grouped by the format vendor-Major-minor_toMinor apache-2.0-7
is good for 2.0.0 to 2.7.5 DISTRIBUTION from apache
if there are different dependencies more/less or @HDFS_VER@ is not equal
to all a new group should be created.
Although group has one name the built and module get the actual distro
version and the built target is
ht-fsbroker-apache-hadoop-${HDFS_VER}-${VERSION}-bundled.jar
The configuration to vendor/grouping is done by the vendor cmake files
at hadoop_distro
and apache-1.0_2 group is configured as follow pattern
elseif (HDFS_VER MATCHES "^2.[0-7]")
## APACHE-HADOOP 2.0.0 to 2.7.5 DISTRIBUTION
set(HDFS_VER_GROUP "apache-2.0_7")
thriftclient is not depends on hadoop sources of any kind while depends
on thriftgen and added with libthrift and available as stand alone jar,
thrift proto version dependant
thriftclient-hadoop-tools were previusly the hypertable/common/hadoop
which are actually stand alone tools dependent on thriftclient
Available cmake ARGUMENTS:
-Dlanguages=php,pl,py2,pypy2,py3,pypy3,java,js,rb
mainly apply when thriftclient should be built as required or skipped
default is try all
-Dfsbrokers=hdfs,ceph,qfs,mapr
which FsBroker to build, required if set.
default is try all
-Dhdfs_vers=apache-2.7.5
(the full distribution format, like apache-2.7.5)
require hdfs to be in fsbrokers param
the follow can work,
-Dhdfs_vers=apache-2.7.5,apache-1.1.0,apache-1.1.1,apache-1.2.1
default is the available hdfs setup
to make a build of only java, target is make MVN_build
For the config
-Dhdfs_vers=apache-2.7.5,apache-1.1.0,apache-1.1.1,apache-1.2.1 with
Thrift available reactor order/inclusion would be as such:
[INFO] Reactor Build Order:
[INFO]
[INFO] Umbrella project for Hypertable
[INFO] Hypertable Common
[INFO] Hypertable apache-hadoop-2.7.5 FsBroker
[INFO] Hypertable apache-hadoop-1.1.0 FsBroker
[INFO] Hypertable apache-hadoop-1.1.1 FsBroker
[INFO] Hypertable apache-hadoop-1.2.1 FsBroker
[INFO] Hypertable Thrift Client - v0.10.0
[INFO] Hypertable Thrift Client - v0.10.0 - Hadoop Tools
The install result is:
-- Installing:
/opt/hypertable/0.9.8.11/lib/java/ht-fsbroker-apache-hadoop-2.7.5-0.9.8.11-bundled.jar
-- Installing:
/opt/hypertable/0.9.8.11/lib/java/ht-fsbroker-apache-hadoop-1.1.0-0.9.8.11-bundled.jar
-- Installing:
/opt/hypertable/0.9.8.11/lib/java/ht-fsbroker-apache-hadoop-1.1.1-0.9.8.11-bundled.jar
-- Installing:
/opt/hypertable/0.9.8.11/lib/java/ht-fsbroker-apache-hadoop-1.2.1-0.9.8.11-bundled.jar
-- Installing:
/opt/hypertable/0.9.8.11/lib/java/ht-thriftclient-0.9.8.11-v0.10.0-bundled.jar
-- Installing:
/opt/hypertable/0.9.8.11/lib/java/ht-thriftclient-hadoop-tools-0.9.8.11-v0.10.0-bundled.jar
-- Installing:
/opt/hypertable/0.9.8.11/lib/java/ht-common-0.9.8.11-bundled.jar
Why bundled, less open-fds by java and less issues with version
managing, while ht-fsbroker depends on ht-common and hadoop-tools on
thriftclient.
Java ThriftClient can run like:
java -cp
/opt/hypertable/0.9.8.11/lib/java/ht-thriftclient-0.9.8.11-v0.10.0-bundled.jar
org.hypertable.thrift.BasicClientTest
v0.10.0 - stands for the include Thrift proto version and the thriftgen
While hadoop distro have a change in ./ht-set-hadoop-distro.sh
change mainly is, the available hadoop FsBrokers are the jar available
in the lib/java by lib/java/ht-fsbroker-*.jar
for example, from the cmake param
-Dhdfs_vers=apache-2.7.5,apache-1.1.0,apache-1.1.1,apache-1.2.1
ht-set-hadoop-distro.sh for usage help will output the followingL:
The currently supported values for <distro> include:
apache-hadoop-1.1.0
apache-hadoop-1.1.1
apache-hadoop-1.2.1
apache-hadoop-2.7.5
ht-java-run.sh undergo a simplification and has the ARG --cp-group
ClassPathGroup which is inclose FsBroker group as thriftclient or other
might not need the fsbroker jar
ht-start-fsbroker remained with hadoop as file-system type unchanged,
while added with the classpath group ht-java-run.sh --cp-group FsBroker
-- without use of pre-generated
(bunded eq 36.1MB)
lib linking start from first target require
-DBUILD_WITH_STATIC=ON/OFF (option to link against static ) -DENABLE_SHARED=ON (Building static libraries by default, with ON shared and static are built with seperate targets) More useful functions in CommonUtil.cmake
…os type but Linux) additional available libs for static linking: libevent.a liblzma.a
- Hypertable.Config.OnFileChange.file
-- a module to Hypertable::Client
methods:
init
create_namespace
exists_namespace
open_namespace
drop_namespace
supporting test HypertableClient-test (ht_client_test.py)
-- ht-package re-organized
- new forlder for tests
- pkg building dir changed from "src" to "hypertable"
- added extension to setup.py
-- get, sets the default value if hasn't name
it is not a must results are after "Waiting for connection to Hyperspace"
- fill up properties of new cfg onto existing
- continue on not has('config') file
- no issues with read-error for cases such as: ./ht_ssh localhost 'echo "sleeping 10"; sleep 10; echo "wokeup"'
- verifying before use hdfsNewBuilder hdfsBuilderConnect - with hdfsGetUsed hdfsGetCapacity
- added "T_of get_item(size_t)" to ValueGuardedVector<T>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
supporting information on the release candidate version 0.9.8.15
https://github.com/kashirin-alex/hypertable/tree/0.9.8.15-rc/built_logs/0.9.8.15
Major subjectives,
- Dynamic Configurations Reload(reconfiguration) at run time
- Support for updated dependencies releases
Version 0.9.8.15:
(2018-12-19)
Version 0.9.8.14:
(2018-08-23)