Actimem http://actimem.com Experts in Wordpress, Php, Java, C#, Oracle and MySql. Sun, 18 Nov 2018 22:49:00 +0000 en-GB hourly 1 https://wordpress.org/?v=4.9.9 83590568 Bash Shell Scripting http://actimem.com/linux/bash-shell-scripting/ http://actimem.com/linux/bash-shell-scripting/#respond Mon, 30 Apr 2018 07:06:31 +0000 http://actimem.com/?p=633 Introduction A shell script is a file with commands that will be executed by the shell. Shell scripts benefit from programming concepts such as if statements, loops, variables and functions. The aim of this article is to provide enough knowledge to comfortably read and write bash shell scripts. Please note this article only looks at bash shell scripts.  You … Continue reading Bash Shell Scripting

The post Bash Shell Scripting appeared first on Actimem.

]]>
Introduction

A shell script is a file with commands that will be executed by the shell. Shell scripts benefit from programming concepts such as if statements, loops, variables and functions. The aim of this article is to provide enough knowledge to comfortably read and write bash shell scripts.

Please note this article only looks at bash shell scripts.  You will be able to use most of the concepts described in this section with other shells, such as C shell, Korn shell, tcsh, zsh or older families of the bourne shell. You however need to be aware there are differences.

Creating a Shell Script

Let’s create our first script:

#!/bin/bash
# A comment
echo "My name is $USER"

The first line of a shell script specifies the shell that will be used to execute the script.  This is done with the sha-bang

#!
  (also known as hash-bang) followed by the shell path. The sha-bang line provides portability, as it ensures your script will always be executed with the same shell regardless of environment configuration and *nix platform.

Before running the script, we need to give it execution permission:

$ chmod u+x basic.sh

And finally we execute it:

$ ./basic.sh
My name is Eduard

Note we need to prefix our script file with

./
  if the current directory is not in the
$PATH
.

Variables

Setting Variables, Exporting and Sourcing

A variable stores a value that can be accessed or changed at a later time. This is the syntax to set variables in bash:

var="a variable value"

A variable will exist until the process ends or it is unset:

unset var

A variable is only known to the shell or script where it is defined. If you want to pass a variable to the child process, you need to export it:

var="a variable value"
export var

or you can use the abbreviated form:

export var="a variable value"

The 

echovar.sh
 script below will help illustrate this behaviour.

#!/bin/bash
echo The var variable is \"$var\"

When the

var
 variable is set in the shell command line but not exported, it is not visible by the 
echovar.sh
 script. Once we export it, it becomes visible:

$ var="a value"
$ ./echovar.sh
The var variable is ""
$ export var
$ ./echovar.sh
The var variable is "a value"

Another source of confusion worth mentioning is when we set variables within a script, and then we expect those variable changes to be available in the parent shell. For example, look at the following 

setvar.sh
 script:

#!/bin/bash
var="a variable value"

If you execute this script from the shell, and then echo the variable from the command line, you will get an empty value. This is because the shell creates a new child process to execute the script, and child processes can never make changes to parent processes.

$ ./setvar.sh
$ echo var=$var
var=

An alternative is to use the

.
 or
source
 commands to execute the script.  With the
.
 and 
source
 commands the shell executes the script in its current process, instead of creating a child process. This means that any changes will be applied directly to the shell process.

$ source ./setvar.sh
$ . ./setvar.sh

Accessing Variables

The simplest way to access a variable is to prefix it with the

$
 sign:

$ var="a value"
$ echo $var
a value

Alternatively, you can also use the

${ }
 form:

$ echo ${var}

You can also include variables inside strings:

$ echo "This is [$var]"
This is [a value]

Parameter Expansion

Parameter expansion is a mechanism to access the value of the variable, but to return an alternative value depending on a test. For example, return a default value when the variable has not been set.

These are the main parameter expansion types:

Parameter Expansion Description
${var:-default}
Expand to default value if the variable is unset or empty
$ var=
$ echo ${var:-"default value"}
default value
$ unset var
$ echo ${var:-"default value"}
default value
${var-default}
Same as
${var:-default}
, but only applies when the variable is unset, and not when it is empty
${var:+alternative}
Replaces the contents of the variable with an alternative value when the variable is set.  It does not apply when the variable is unset or empty.
$ var="a value"
$ echo ${var+"another value"}
another value
$ unset var
$ echo ${var:+"another value"}

$ var=
$ echo ${var:+"another value"}
${var+alternative}
Same as
${var:+alternative}
, but also applies when the variable is empty.

This is to say, we  use the alternative value when the variable is either set or empty, but not when it is unset.

${var:=default}
Expands and sets the variable to a default value if unset or empty:

$ var=
$ : ${var="a value"}
$ echo ${var}
a value
$ unset var
$ : ${var:="a value"}
$ echo ${var}
a value

${var=default}
Same as
${var:=default}
 , but  only applies when the variable is unset, and not when it is empty.
${var:?message}
If variable is unset or emtpy, it prints the message to standard error and exits the script with status of 1.

$ cat test.sh
#!/bin/bash
: ${1?You need at least one argument}
echo Your argument=$1

$ ./test.sh
./test.sh: line 2: 1: You need at least one argument
$ echo $?
1

$ ./test.sh  "value"
Your argument=value
$ echo $?
0

${var?message}
Same as
${var:?message}
, but only applies when the variable is unset, and not when it is empty.
${#var}
Returns the length of the variable value

$ var=
$ echo ${#var}
0
$ var="a value"
$ echo ${#var}
7

 ${var:offset:length}
Returns a substring of the the variable value.

The first character offset is 0.

$ var=0123456789
$ echo ${var:0:5}
01234
$ echo ${var:3:5}
34567
$ echo ${var:8:5}
89

If the offset is not provided, it then assumes it is 0.

$ echo ${var::5}
01234

You can also use a negative offset to count from the end of the string. Note that you need to leave a space between the

:
 and the negative sign
-
  as otherwise if would conflict with
${var:-default}

$ echo ${var: -4:2}
67

If you do not provide the length, it then assumes it is until the end of the string:

$ echo ${var:3}
3456789

${var//pattern/string}
Replaces all instances of the pattern in the variable value with the string.  The regular expression is the same as pathname regular expresion. That is,
?
 matches any character, and
*
 matches zero or more characters.

$ var="this is a value"
$ echo ${var//is/XX}
thXX XX a value
$ echo ${var//t?is/XX}
XX is a value
$ echo ${var//s*v/XX}
thiXXalue
$ echo ${var//v*/}
this is a

${var^}
Replaces the first character to upper-case

$ var="this is a value"
$ echo ${var^}
This is a value

${var^^}
Replaces all characters to upper-case

$ var="this is a value"
$ echo ${var^^}
THIS IS A VALUE

${var,,}
Replaces all characters to lower-case

$ var="THIS IS VALUE"
$ echo ${var,,}
this is value

Arrays

Arrays are programming structures that store multiple values under the same variable. Bash supports two types of arrays: integer-indexed arrays and associative arrays.

Integer-Indexed Arrays

Integer-indexed arrays are zero based and do not need to be contiguous. This is to say, you can store a value at index 0, and another at index 30.

$ array[0]=element-0
$ array[30]=element-30

You can reference any value in the array as follows:

$ echo $array[0]
element-0
$ echo ${array[30]}
element-30

You can return all values in the array using the indexes

[*]
 and
[@]
 .

$ echo ${array[*]}
element-0 element-30
$ echo ${array[@]}
element-0 element-30

When the array reference is double-quoted,  

[]
 and
[@]
  have a different meaning.  
${array[]}
 returns a string with each element separated by space, while    
${array[@]}
 returns a separate string for each element.

$ printf "%s\n" "${array[*]}"
element-0 element-30
$ printf "%s\n" "${array[@]}"
element-0
element-30

You can return print the number of values in the array as follows:

$ echo ${#array[*]}
2
$ echo ${#array[@]}
2

You can also initialise arrays with a single command:

$ array=( element-0 element-1 element-2 )
$ echo ${array[@]}
element-0 element-1 element-2

And you can also append to the end of the array using

+=
 :

$ array=( element-0 )
$ array+=( element-1 element-2 )
$ echo ${array[@]}
element-0 element-1 element-2

Associative Arrays

Associative arrays are referenced using strings, similar to maps in other programming languages.

Before you can use a variable as an associative array, you need to declare it:

$ declare -A map

After this, you can add any items to the array:

$ map["key1"]=value1
$ map["key 2"]=value1
$ map[key3]=value3

You can reference to values in the array as usual:

$ echo ${map["key1"]}
value1
$ echo ${map["key 2"]}
value2
$ echo ${map[key3]}
value3

You can also return all values of the array using the 

[*]
 and
[@]
  index references.  When they are double-quoted, they have same behaviour as described in the integer-indexes arrays.

$ echo ${map[*]}
value1 value2 value3
$ echo ${map[@]}
value1 value2 value3

If you want to return the keys, you can use the

${!map[@]}
  notation:

$ printf "%s\n" "${!map[*]}"
key1
key 2
key3

Positional Parameters

Positional parameters reference the arguments passed to the script by the shell. These arguments can be referenced as

$1
 ,
$2
 ,
$3
,…

For example, let’s use the following

params.sh
 script

#!/bin/bash
echo $1
echo $2
echo $3

This is the result when executing it from the shell:

$ ./params.sh param1 param2 param3
param1
param2
param3

There are several types of positional parameters:

Type Description
$1, $2, $3,...
Reference the first, second, third… argument provided by the shell
${10}, ${11},...
Reference large number of arguments
$0
The script name
$#
The number of arguments
$*
If not quoted, it returns all parameters as separate strings.

If quoted, it returns all parameters in the same string, each separated by a space.

$ cat ./params2.sh
#!/bin/bash
printf "%s\n" ${*}
printf "--\n"
printf "%s\n" "${*}"

$ ./params2.sh param1 param2 param3
param1
param2
param3
--
param1 param2 param3

$@
If not quoted, it returns all parameters as separate strings (same as $*)

If quoted,

"$@"
 returns the equivalent to
"$1" "$2" "$3" ...
, that it, separate strings.

$ cat ./params3.sh
#!/bin/bash
printf "%s\n" ${@}
printf "--\n"
printf "%s\n" "${@}"

$ ./params3.sh param1 param2 param3
param1
param2
param3
--
param1
param2
param3

Shift

The

shift
 command removes the first positional parameter, and shifts the remaining parameters one place to the left.

For example, look at the following script where we shift the positional parameters twice:

#!/bin/bash
echo $@
shift
echo $@
shift
echo $@

This will be the output when passing 4 parameters to the script:

$ ./params.sh param1 param2 param3 param4
param1 param2 param3 param4
param2 param3 param4
param3 param4

You can also shift several parameters at a time:

shift 3

and you can also remove all parameters with

$#
:

shift $#

Exit Status

All scripts return an exit status once they complete their execution. By convention, an exit status of 0 means success, and anything else means failure. The exit status is an integer between 0 and 255.

You can capture the exit status of the previous command with the

$?
 special variable:

$ test 1 -eq 2
$ echo $?
1
$ test 1 -eq 1
$ echo $?
0

The

exit
 command is used to exit the script and return an exit status to the parent process.

$ cat run_with_exit.sh
#!/bin/bash
echo "let's run a script!"
exit 2
$ ./run_with_exit.sh
let's run a script!
$ echo $?
2

If the script does not have an exit command, then it returns the exit status of the last command that executed.

$ cat run_no_exit.sh
#!/bin/bash
test 1 -eq 2
$ ./run_no_exit.sh

$ echo $?
1

Comparisons: test, [ and [[ Commands

The

test
 command evaluates a conditional expression and returns 
 when true, or 
1
 when false.  The example below uses
test
 to check if the parameter passed to a script is 5.

$ cat test.sh
#!/bin/bash
if test $1 -eq 5
then
   echo "Parameter is 5 as expected!"
else
   echo "Please try again!"
fi

$ ./test.sh 5
Parameter is 5 as expected!

$ ./test.sh 1
Please try again!

The square bracket command 

[
 is a replacement for the test command that feels more natural when used in
if/then
 statements.  The
[
  command supports the same conditions as the test command.  The syntax is  
[ expression ]
.  The script below is equivalent to the previous one.

$ cat test.sh
#!/bin/bash
if [ $1 -eq 5 ]
then
   echo "Parameter is 5 as expected!"
else
   echo "Please try again!"
fi

The double square bracket command

[[
 is an extension over the
[
  command. The 
[[
  command supports all conditions that 
[
 supports, plus string pattern matching.  The syntax is
[[ expression ]]
 . The script below uses pattern matching to test whether the parameter starts with the letter
a
.

$ cat test_pattern_matching.sh
#!/bin/bash

if [[ $1 == a* ]]
then
   echo 'The input starts with the letter a'
else
   echo 'bad input!'
fi

$ ./test_pattern_matching.sh actual
The input starts with the letter a

$ ./test_pattern_matching.sh zip
bad input!

It is important to remember that both

[
 and
[[
 are commands. As any other command, you need to separate the command name from its parameters by a space. So you always need to leave a space after the opening square bracket, and another space before the closing square bracket.

The

test
[
 and
[[
 commands support the following type of expressions:

  • Numeric comparisons
  • String comparisons
  • File comparisons
  • Pattern matching comparisons (only
    [[
     )

Numeric Comparisons

These are the expressions used to compare integers. Valid for all

test
[
 and
[[
 commands.

Operation Description
a -eq b
a is equal to b
a -ne b
a is not equal to b
a -gt b
a is greater than b
a -ge b
a is greater than or equal to b
a -lt b
a is less than b
a -le b
a is less than or equal to b

String Comparisons

These are the expressions used to compare strings. Valid for all

test
[
 and
[[
 commands. Note the differences between
[
  and
[[
 .

Operation Description
str1 = str2
str1 is equal to str2

Note that string comparison is just one equals symbol (

=
).

[[
  uses two equal symbols (
==
) for pattern matching.

 

str1 != str2
str1 is not equal to str2
str1 < str2
str1 is less than str2

test
  and
[
  use standard ASCII ordering.

[[
 uses the lexicography of the current locale.
str1 > str2
str1 is greater than str2.
-n str
length of string is not zero
-z str
length of string is zero

File Comparisons

These are the expressions used to check the file status of files and directories. Valid for all

test
[
 and
[[
 commands.

Operation Description
-a file
True if file exists
-e file
True if file exists
-d file
True if file exists and is a directory exists
-f file
True if file exists and is a regular file
-r file
True if file exists and is readable
-w file
True if file exists and is writable
-x file
True if file exists and is executable
-s file
True if file exists and is not empty
file1 -nt file2
True if file1 is newer than file2
file2 -ot file2
True if file1 is older than file2

Below an example script that checks whether the file provided in the command line is a directory, regular file and whether it can be executed.

$ cat numeric_test.sh
#!/bin/bash

if [ -d $1 ]
then
  echo "$1 is a directory"
fi

if [ -f $1 ]
then
  echo "$1 is a regular file"
fi

if [ -x $1 ]
then
  echo "$1 is an executable"
fi

Pattern Matching

Below the pattern matching expressions. Only supported by the

[[
 command.

Operation Description
str == pattern
True if str matches the pattern.

It uses the shell script pattern matching rules (e.g. only

*
  and
?
  are supported)

 

str != pattern
True if str does not match the pattern

It uses the shell script pattern matching rules (e.g. only

*
  and
?
  are supported)
str =~ pattern
True if str matches the pattern.

It uses extended regular expressions (it also supports

+
 )

 

Command Substitution

pg 277

 

 

Bibliography

The post Bash Shell Scripting appeared first on Actimem.

]]>
http://actimem.com/linux/bash-shell-scripting/feed/ 0 633
Java NIO Buffer http://actimem.com/java/nio/buffer/ http://actimem.com/java/nio/buffer/#respond Sat, 02 Sep 2017 11:08:52 +0000 http://actimem.com/?p=717 This article explains key concepts of a Java NIO buffer: position, limit, capacity and flipping, and the difference between direct and non-direct buffers.

The post Java NIO Buffer appeared first on Actimem.

]]>
Java NIO buffers are holders of information in an array-type structure. At its heart, NIO processing is about moving data in and out of buffers. Unlike traditional Java I/O that uses separate input and output streams, a Java NIO buffer is used for both read and writing. Buffers are abstractions that allow exchanging of data.

Working with the java.nio.Buffer API, however, is not straightforward as you need to understand low-level concepts such as position, limit, capacity and flipping. This article provides a detailed tutorial of how to work with the java.nio.Buffer API and sub-classes.

The Java NIO Buffer hierarchy

The java.nio.Buffer class is the high-level abstraction of a Java NIO buffer. This class defines the operations common to all buffer types, such as current position, limit, capacity, flipping, marking, and rewinding.

There are several subclasses of java.nio.Buffer, one for each primitive type:

Additionally, there is also a MappedByBuffer class that extends from ByteBuffer that is used to work with direct buffers. More on direct vs non-direct buffers later.

ByteBuffer is the most important buffer type, as operating systems work at byte level. The other buffer types provide a convenience interface to work with primitives such as integers, doubles or chars.  But when interacting with the operating system, we need to use byte buffers.

Creating Buffers

New buffers are created by either allocation or wrapping.

With allocation, you just specify the capacity of the buffer, and let the NIO framework create the internal structures to store the data. The current Java implementation use a backing array of the buffer type, but this is not guaranteed.

This is how you allocate a byte buffer of 100 bytes.

ByteBuffer byteBuffer = ByteBuffer.allocate(100);

And this is how you create a char buffer of 100 chars:

CharBuffer charBuffer = CharBuffer.allocate(100);

With wrapping, you need to provide a backing array:

byte[] backingArray = new byte[100];
ByteBuffer byteBuffer = ByteBuffer.wrap(backingArray)

There are two types of byte buffers, direct and non-direct buffers.  Direct buffers are mapped outside of the Java heap and can be accessed by the operating system directly, thus they are faster than non-direct buffers backed by arrays in the Java heap.  Direct buffers are created with the allocateDirect() method:

ByteBuffer byteBuffer = ByteBuffer.allocateDirect(100);

Position, Limit and Capacity

The key to understanding Java NIO buffers is to understand buffer state and flipping. This section introduces the three main properties of buffer state:  position, limit and capacity.

Capacity is the maximum number of data items that the buffer can hold. For example, if you create a buffer with the backing array new byte[10], then the capacity is 10 bytes.  The capacity never changes after buffer creation.

Limit is the zero-based index that identifies the first data item that should not be read or written. Limit determines the data that can be read from the buffer.  Data between zero and limit (exclusive) is available for reading. Data items between limit (inclusive) and the capacity index are garbage.

Position is the zero-based index that identifies the next data item that can be read or written. As you read from or write into the buffer, the position index increases.

Java NIO Buffer state: position, limit and capacity

The following invariant must apply at all times:

0 <= position <= limit <= capacity

The java.nio.Buffer class provides generic methods to access the state:

int position()
int limit()
int capacity()

There are also methods to set the position and the limit. Please note that we cannot change the buffer capacity after creation:

Buffer position(int newPosition)
Buffer limit(int newLimit)

There is also a remaining() method to calculate the number of remaining data items available for consumption.  Remaining is calculated as limit() – position().

int remaining()
boolean hasRemaining()

Reading and Writing Data

Each Buffer implementation provides several get and put methods to read from and write into the buffer.

ByteBuffer, for example, provides the following methods to read and write bytes:

byte get()
ByteBuffer put (byte b)

CharBuffer provides methods to work with chars.

char get()
ByteBuffer put (char c)

The position property plays a central role when reading from or writing into the buffer. The get() and put() methods read and write data for the index pointed by the current position property.  Then, the position index increases ready for the next operation.

NOTE: The remaining of this article will focus on ByteBuffer, as it is the most important Buffer implementation given the operation system works at byte level.

Buffer Life cycle: Fill, Flip, Drain, Clear

Java NIO buffers are structures that enable the exchange of data, and they are used for both reading and writing.  Conceptually, a Java NIO buffer has two modes of operation:

  • Filling mode – a producer writes into the buffer
  • Draining mode – a consumer reads from the buffer

In the typical life cycle of a Java NIO buffer, the buffer is created empty ready for a producer to fill it up with data. The buffer is in filling mode.  After the producer has finished writing data, the buffer is then flipped to prepare it for draining mode. At this point, the buffer is ready for the consumer to read the data.  Once done, the buffer is then cleared and ready for writing again.

Java NIO Buffer Life cycle

Fill the Buffer

Data is written into a Java NIO buffer using the put() method. The following code illustrates how to fill the buffer.

// Create the buffer
ByteBuffer buffer = ByteBuffer.allocate(6);

// Add three bytes
buffer.put((byte) 10);

// Add another byte
buffer.put((byte) 11).put((byte) 12).put((byte) 13);

First, we create a buffer with capacity 6. The buffer is now empty ready to be filled. The limit and capacity properties are pointing at index 6, and position is pointing at 0.

Java NIO Buffer after allocation

The first put() writes a byte into index 0, and then increased the position to index 1. Then we add three more bytes into the buffer. After this, the position is pointing at index 4, with 2 remaining data slots in the buffer.

Java NIO Buffer after put

Flip the Buffer

Once the producer has finished writing data to the buffer, we need to flip it so that the consumer can start draining it.

buffer.flip();

Why do we need flipping? If we did not flip, the get() method would read data from the current position. In our example, we would be reading data from index 4, which is a position without data.

Java NIO Buffer after flip()

flip() is a method that prepares the buffer to retrieve its contents. It sets the limit property to the current position to mark the area of the buffer with data content, and the position is reset back to 0 so the get() operation can start consuming the data from the beginning of the buffer.

In practical terms, flip() is equivalent to the following

buffer.limit(buffer.position()).position(0);

Drain the Buffer

After the buffer has been flipped, we are ready to start reading with the get() method:

System.out.println(buffer.get());

This reads byte 10 from index 0, and increases the position to index 1.

Java NIO Buffer: Get data from buffer

We can also drain the buffer completely by checking the hasRemaining() method until we reach the buffer limit:

while (buffer.hasRemaining())
    System.out.println(buffer.get());

This is how the buffer looks like after draining:

Java NIO Buffer: Get data from buffer

Clear the Buffer

Once the buffer has been drained, the next step is to prepare the buffer for filling again.  This is done with the clear() method.

buffer.clear();

The clear() method sets the position back to 0 and the limit to same value as capacity. Please note clear() does not remove the data from the buffer, it just changes position and limit.

Java NIO Buffer clear() method

The clear() method is equivalent to the following:

buffer.position(0).limit(buffer.capacity());

You might wonder why we didn’t flip() the buffer instead of clear().  This is because flip() changes the limit property to the current position, thus it would not allow to fill the buffer to its full capacity.

Reading and Writing Data in Bulk

The buffer API provides methods to read and write data in bulk using arrays.

ByteBuffer has two bulk methods for put():

ByteBuffer put(byte[] data)
ByteBuffer put(byte[] data, int offset, int length)

The first put() method writes the full content of the data array into the buffer starting at the current position. The position will be incremented by the length of the array. The second put() method writes the contents of the array starting at the offset position, and copying length bytes.  If we attempt to copy more bytes than remaining bytes in the buffer, we will get a BufferOverflowException.

Similarly, the ByteBuffer has two counterparts for reading in bulk:

ByteBuffer get(byte[] data)
ByteBuffer get(byte[] data, int offset, int length)

The bulk get() method reads the contents of the buffer into the data array starting at the current position, until filling up the array completely.  Note that if the array provided is larger than the number of remaining bytes in the buffer, a BufferUnderflowException exception is thrown.

The following code illustrates how to read and write in bulk:

// Allocate the buffer
ByteBuffer buffer = ByteBuffer.allocate(6);

// Put data in bulk
byte[] data = new byte[] {(byte) 10, (byte) 11, (byte) 12};
buffer.put(data);

// Flip the buffer ready for draining
buffer.flip();

// Read data in bulk
byte[] readData = new byte[buffer.remaining()];
buffer.get(readData);

Direct vs Non-Direct Buffers

There are two types of buffers, direct and non-direct.

A non-direct buffer is backed by an array in the Java heap. The operating system, however, does not copy data in or out of this array directly.  Instead, it uses an intermediate buffer outside of the Java heap. This intermediate buffer is needed because the operating system cannot efficiently access objects in the Java heap, as objects might not be optimally page aligned or might be relocated by the Garbage Collector.

Non-direct buffers are created with the following allocate() method:

ByteBuffer byteBuffer = ByteBuffer.allocate(100);

As an example, imagine an application needs to read data from a file in local disk. It will make a call to the operating system, which in turn will instructs the disk controller to copy bytes from storage to the intermediate buffer.  The data from this operating system buffer is then copied to the Java heap buffer. The application can then access data from this buffer via a java.nio.ByteBuffer instance.

Java NIO Non-Direct Buffer

A direct buffer is a byte buffer that access directly the memory used by the operating system to store the data from the I/O resource.  This buffer is outside of the reach of the Garbage Collector.

Direct buffers are created with the following allocateDirect() method:

ByteBuffer byteBuffer = ByteBuffer.allocateDirect(100);

Direct buffers are more efficient as I/O operations are performed directly on the buffer without the need of copying the information into memory first. However, creating a direct buffer is an expensive operation, and might even trigger a full Garbage Collection. Direct buffers are usually best suited when working with long-lived and large buffers, although performance gain should be measured before committing to using direct buffers.

Summary

This article has explained in great length the key concepts of NIO buffers. A Java NIO buffer is an array-like structured with three properties: position, limit and capacity.  The same buffer is used for both producers and consumers, thus conceptually we need to differentiate between two modes of operation: filling and draining modes.  To change from one mode to another we use the flip() and clear() methods. Finally, we explored the differences between direct and non-direct buffers.

Bibliography

The post Java NIO Buffer appeared first on Actimem.

]]>
http://actimem.com/java/nio/buffer/feed/ 0 717
Introduction to Java NIO http://actimem.com/java/nio/introduction-java-nio/ http://actimem.com/java/nio/introduction-java-nio/#respond Fri, 25 Aug 2017 21:39:26 +0000 http://actimem.com/?p=703 This article provides an overview of Java NIO and NIO.2, and will be the foundation for later articles that will explore Java networking programming.

The post Introduction to Java NIO appeared first on Actimem.

]]>
This article provides an overview of Java NIO and NIO.2, and will be the foundation for later articles that will explore Java networking programming using NIO.

A brief history of Java NIO

The capability to work with files and byte streams has been available since JDK 1.0. This first version of the java.io package, however, did not fully support character streams. This was added to JDK 1.1 with the reader and writer classes, and the introduction of character encodings. This stream API, however, was blocking and did not scale well. A web server handling thousands of simultaneous requests, for example, needed a separate thread for each connection.

The next milestone in streaming processing was introduced in Java 4 with the new Java NIO API (JSR 51). This API promised to provide fast and scalable I/O operations by taking advantage of non-blocking I/O advancements to operating systems. Buffers, Channels and Selectors in the java.nio package made the core of this API. Additionally, it also provided regular expressions for efficient parsing, printf-style formatting, and Charset encoders and decoders

Java 7 enhanced the Java NIO APIs further (JSR-203) by adding efficient file handling, a new file system interface, asynchronous I/O, support for socket binding and multicasting. The makeover of this API was so substantial that was rebranded as Java NIO.2.

Java 8 enhanced the NIO.2 API further by adding support for the new java.util.stream API.

Buffers

Buffers are the workhorse of Java NIO. At its heart, NIO processing is about moving data in and out of buffers. The java.nio.Buffer class is the main abstraction, while java.nio.ByteBuffer is the main implementation class, representing a buffer containing bytes.

There are two types of buffers, direct and non-direct buffers.

A non-direct buffer uses an intermediate buffer managed by the operating system.  This intermediate buffer is needed because the operating system cannot efficiently access buffers in the Java heap.  All objects in the Java Heap are managed by the VM and might change memory location or not be optimally page aligned.

As an example, imagine an application that needs to read data from a file in local disk. It will request to read this data from the operating system, which in turn will instruct the disk controller to copy bytes from storage to the intermediate buffer.  The data from this buffer will then be copied to the Java heap buffer. The application can then access data from this buffer via a java.nio.ByteBuffer instance.

Java NIO Non-Direct Buffer
Java NIO Non-Direct Buffer

A direct buffer is one that removes this intermediate buffer by moving the underlying buffer memory outside of the Java heap.  As the memory address is fixed for the lifetime of the buffer, and outside of the reach of the Garbage Collector, the operating system can safely access it. While these buffers are more efficient when reading and writing, they are also more expensive to create and destroy.

Channels

A channel represents a connection to a resource that supports I/O operations such as reading and writing from files, network sockets, hardware devices or other applications.  Channels are used in conjunction with buffers to achieve efficient streaming of data. While channels coordinate with the operating system to access the resource, buffers are used to transfer and hold the data.

Channels are defined in the java.nio.channel package, and abstracted by the java.nio.channel.Channel interface.

These are the three main channel implementations:

Selectors

Selectors enable the efficient monitoring of many Java NIO channels and recognise when they are ready to transmit or receive data.  This allows a single thread to manage large number of channels, thus reducing the system load by reducing thread context overhead and thread memory utilisation.

Selectors make use of a modern operating system feature called non-blocking mode access. Non-blocking access allows the operating system to check the readiness of a stream without blocking. The selector can then instruct the operating system to monitor a group of channels and get notified once one or more are ready for processing.

The main Selector class in the java.nio.channel.Selector, and are used mainly for network servers to listen from multiple socket connections.

NIO Selector
NIO Selector

Summary

This article has presented a brief history of Java NIO and introduced the main concepts of the Java NIO API that enable efficient I/O streaming: Buffers, Channels and Selectors.  Later articles will dive into these concepts and explain how to use the Java NIO for networking programming.

Bibliography

The post Introduction to Java NIO appeared first on Actimem.

]]>
http://actimem.com/java/nio/introduction-java-nio/feed/ 0 703
Find command in Linux http://actimem.com/linux/find/ http://actimem.com/linux/find/#respond Sat, 03 Sep 2016 18:12:12 +0000 http://actimem.com/?p=598 The find tool searches files and directories that match a certain criteria, and executes actions on those files. Find is one of the most useful programs you can have in your Linux toolset. This article explains the foundations of the find tool so that you can use it to its full potential.

The post Find command in Linux appeared first on Actimem.

]]>
Introduction

The find tool searches files and directories that match a certain criteria, and executes actions on those files. Find is one of the most useful programs you can have in your Linux toolset. This article explains the foundations of the find tool so that you can use it to its full potential.

Find Command-Line Syntax

This is the basic command-line syntax of find:

find [paths] [expression]

The

[paths]
 option defines one or more paths where find will start the search.  If no paths are provided, then
.
 is assumed (current directory)

The expression describes the files that need to be selected (tests) and what to do with those files (actions). If no expression is provided, then the

-print
  action is used.

There are four types of expressions:

Action return true or false depending on the success or failure status returned after executing the action.

Tests Return true or false depending on a test condition on the file, such as name mathces an expression, it is a directory or it was modified 2 days ago.
Actions Actions are expressions that have side effects, such as printing the file name to standard output, or executing an arbitrary program.
Operators Operators are logical operations that join together expressions.  Examples are
-a
 for logical AND,
-o
 for logical OR,
-!
 for logical NOT.

If no operator provided, then

-a
  is assumed.
Global Options Global options configure the operation of tests and actions

 Tests

Tests are expressions that return true or false depending on a condition on the file, such as file name matches a regular expression, file type is a directory, or file was modified 2 days ago. These are the most useful tests of the find tool (for a full list, check the find manual):

-name pattern
Finds files names that match the specified pattern.

It uses the same pattern matching rules as the shell, so you can use

*
,
?
 and
[ ]
 .

It strips out the preceeding directories, so

-name
 will not match directories even if you use  
/
 . For that, you should use 
-path
.

$ find . -name "*txt"

-path pattern
Finds files whose path match the pattern provided.

The metacharacters 

*
,
?
 and
[ ]
  do not treat the directory separator 
/
 any differently. You could therefore use
"./dir*dat"
 to match
"./dira/file.dat"

$ find . -path "./dir*dat"

All path names start with a leading

./
.

$ find . -path "./dir2/*"

-regex pattern
Finds files whose path matches the provided regular expression.

By default it uses the Emacs Regular Expressions. This can be changed with

-regextype
-cmin n
Finds files whose contents or attributes where modified exactly
n
 minutes ago. Use
-n
 to specify less than
n
  minutes ago. Use
+n
 to specify more than n minutes ago.
-mmin n
Finds files whose contents where modified exactly
n
  minutes ago.  As
-cmin
, you can also use
-n
 and
+n
.
-amin n
Finds files accessed exactly 
n
  minutes ago.  As
-cmin
, you can also use
-n
 and
+n
.
-ctime n
Finds files whose contents or attributes where modified
n*24
 hours ago.  Use
-n
 to specify less than
n
  days ago. Use
+n
 to specify more than
n
 days ago.
-mtime n
Finds files whose contents where modified
n*24
 hours ago. As
-ctime
, you can also use
-n
 and
+n
.
-atime n
Finds files accessed 
n*24
 hours ago. As
-ctime
, you can also use
-n
 and
+n
-newer file
Finds files that were modified more recently than the file provided.
-empty
Finds files or directories that are empty
-size n[bckMG]
Finds files of a particular size. You can use
+n
 and
-n
 to indicate file sizes more than and less than. The suffix indicates the unit.

c byte
b 512-byte blocks (default)
k kilobytes (1024 bytes)
M Megabytes
G Gygabytes

To find all image files bigger than 1 Megabyte

$ find . -name "*jpg" -size +1M

-type c
Finds files of a certain type. These are the most useful values of type 
c
 :

f regular file
d directory
l symbolic link

To find all directories and files respectively:

$ find . -type d
$ find . -type f

Actions

Actions are operations on the files that the test conditions selected.  Action return true or false depending on the success or failure status returned after executing the action.

-print
Prints the full name of the matching file to the standard output, followed by new line.

This is the default action if no other actions are provided.

-ls
Lists the current file same as
ls -dils
-delete
Deletes the selected file
-exec command ;
Executes the command. The string
{}
 is replaced by the current file name. The command ends in a
;
, although usually this semicolon needs to be escaped
\;

find . -exec file {} \;
-prune
Skip a directory and its sub-directories.

Operators

These are the find tool operators:

( expr )
Parenthesis are used to force precedence.

You will probably need to use 

\(...\)
  instead of
(...)
  if you run from the shell
! expr
! expr
 True if
expr
  is false.
expr1 -a expr2
True if both
expr1
 and
expr2
 are true.

expr2
 is only evaluated if
expr1
 is true.

-a
 is the default operator, so
expr1 expr2
 is equivalent to
expr1 -a expr2
 .
expr1 -o expr2
True if either 
expr1
 or 
expr2
 are true.

expr2
 is only evaluated if
expr1
 is false.

Global Options

Global options always return true and should appear after the paths:

-depth
Processes the directory files before the directory itself.

The

-delete
 action implies
-depth
.
-maxdepth levels
Descends at most the number of levels.

-maxdepth 0
  means only apply the action on the starting directory
-mindepth levels
Ignores all files less than the number of levels.

-mindepth 1
  means process all files but the ones in the starting directory.

Basic Usage

The most basic usage of find is to list the name of all files of the current directory, and all its sub-directories:

$ find
.
./dir1
./dir1/dira
./dir1/dira/dirA
./dir1/dira/dirA/fileA.dat
./dir1/dira/filea.dat
./dir1/dira/fileaaa.dat
./dir1/dirb
./dir1/dirb/fileb.dat
./dir1/file1.dat
./dir1/file1.txt
./dir2
./dir2/emptydir
./dir2/file2.dat

By default, if no path is provided, then it assumes

.
  (current directory).  The following command is equivalent:

$ find .

Also, find uses 

-print
  if no other action is provided. The below command is once again equivalent:

$ find . -print

You can also use

-name
 to match files by name.  You can use the same pattern matching expressions as the shell, such as 
*
,
?
 and
[ ]
 :

$ find . -name "*.txt"
./dir1/file1.txt

To find only regular files, you can use the

-type f
  parameter:

$ find . -type f -name "*dat"
./dir1/dira/dirA/fileA.dat
./dir1/dira/filea.dat
./dir1/dira/fileaaa.dat
./dir1/dirb/fileb.dat
./dir1/file1.dat
./dir2/file2.dat

Or the

-type d
  parameter is you want to file directories only:

$ find . -type d -name "dir?"
./dir1
./dir1/dira
./dir1/dira/dirA
./dir1/dirb
./dir2

You could also provide multiple paths to search for all their files and directories:

$ find dir1/dira dir2 -type d
dir1/dira
dir1/dira/dirA
dir2
dir2/emptydir

Finding Files By Name

The

-name pattern
  expression selects files based on their file name.

$ find . -name "*txt"

The pattern matching rules are the same ones used in the shell, so you can use

*
,
?
 and
[ ]
 to match file names:

$ find . -name "dir??.txt"
$ find . -name "dir[a-z].*"

Also, it should be noted that

-name
 only tests the file name, thus the leading directories are removed prior to evaluating the pattern expression.  This means that if you use  
/
 to match a parent directory, 
-name
 will never work.  You should consider using
-path
  instead.

Finding Files By Path

The

-path pattern
  expression selects files based on their full path.

$ find . -path "*dira*dat"
./dir1/dira/dirA/fileA.dat
./dir1/dira/filea.dat
./dir1/dira/fileaaa.dat

You should note the metacharacters 

,
?
 and
[ ]
  do not treat the directory separator 
/
 any differently. You could therefore use
"./dirdat"
 to match across directories:

$ find . -path "./dir*dat"
./dir1/dira/dirA/fileA.dat
./dir1/dira/filea.dat
./dir1/dira/fileaaa.dat
./dir1/dirb/fileb.dat
./dir1/file1.dat
./dir2/file2.dat

You should also notice that all paths start with a leading

./
.  For example, the following path will match nothing:

$ find . -path "dir2/*"

If you want to match the contents of

dir2
, you need to use the leading
./
:

$ find . -path "./dir2/*"
./dir2/emptydir
./dir2/file2.dat

Finding Files By Time

The

-ctime n
 , 
-mtime n
 and 
-atime n
 operations find files that were created, modified or accessed at some point in the past.

Understanding how 

-ctime
 , 
-mtime
 and
-atime
 reference time is tricky.   
-mtime n
 means files whose contents or attributes where modified between 
n*24
 hours ago and the preceeding 24 hour period.  So for example, 
-mtime 0
  selects files that were modified in the last 24 hours.  
-mtime 1
  selects files modified between 24 and 48 hours ago.  
-mtime +1
  selects files modified more than 48 hours ago, and 
-mtime -1
 selects files modified within the last 24 hours.

The picture below illustrates how 

-ctime
 , 
-mtime
 and 
-atime
 reference time:

How -mtime and -ctime reference time
How -mtime and -ctime reference time

The example below will help clarify this even further.  Imagine the time now is 3 Sep 2016 18:00, and we have four files updated at 17:00 today (file1.txt), another at the same time yesterday (file2.txt), then on 1 Sep (file3.txt), and finally on 31 Aug (file4.txt).

$ date
03 Sep 2016 18:00:00

$ touch -t 201609031700 file1.txt
$ touch -t 201609021700 file2.txt
$ touch -t 201609011700 file3.txt
$ touch -t 201608311700 file4.txt

The table below summarises the outputs of 

-mtime
 (similar outputs for 
-ctime
  and  
-atime
):

Command Files selected
find . -type f -mtime 0
file1.txt
find . -type f -mtime 1
file2.txt
find . -type f -mtime 2
file3.txt
find . -type f -mtime 3
file4.txt
find . -type f -mtime +0
file2.txt

file3.txt

file4.txt

find . -type f -mtime -1
file1.txt
find . -type f -mtime +1
file3.txt

file4.txt

find . -type f -mtime +0 -mtime -3
file2.txt

file3.txt

Executing Commands

The

-exec
  action is useful to execute commands with the files you find.  This is the basic usage:

$ find . -exec echo {} \;

-exec
 is able to execute any command you like.  The file that needs to be executed can be referred to with
{}
 .  The exec command ends with the characters
\;
.

A common usage us

-exec
  is to look for files and then grep them looking for contents:

$ find . -name "*.java" -exec grep Customer {} \;

Also, it should be noted that

-exec
 is an expression that can return either true or false depending on the result of the execution.  For example, the
isdir.sh
 script returns 0 if the file provided is a directory, and 1 if not.

$ cat isdir.sh
#/bin/bash
if [ -d $1 ]
then
   exit 0
fi
exit 1

We could then use this script to print all directories.

$ find . -exec ./isdir.sh {} \; -print
.
./dir1
./dir1/dira
./dir1/dira/dirA
./dir1/dirb
./dir2
./dir2/emptydir

You could also use

xargs
  to execute a command.  The difference is that
-exec
  executes one command per file, while
xargs
 might use multiple files for each execution.

find . | xargs echo

Advanced Usage: Expressions

Find is a tool with two edges.  It has a basic and commonly used syntax  to search for files – the use case of find-, but also supports a more advanced syntax that makes it a truly powerful tool in your Linux toolbox.

The three concepts below are key to understanding the find tool:

  • It iterates over all files and sub-directories
  • For each file,  it applies the expressions provided.  Expressions are evaluated left to right, and stop as soon as evaluate to false.
  • All test conditions, operations and actions are expressions. They evaluate to either true or false.

Let’s start by looking at the following basic find command:

$ find . -name "*txt" -print
./dir1/file1.txt

The above command iterates over all files and sub-directories of the current directory. For each file, it first checks whether it name matches the

"*.txt"
  expression. If it matches, it then prints it out.

Note that expressions are evaluated left to right. If we had first used

-print
  and then
-name "*txt"
 , we would have got the nasty surprise that all files were printed out. This is because we first print the file and then we filter them out.  The filtering, of course, comes too late as we have already evaluated 
-print
 .

$ find . -print -name "*txt"
.
./dir1
./dir1/dira
./dir1/dira/dirA
./dir1/dira/dirA/fileA.dat
./dir1/dira/filea.dat
./dir1/dira/fileaaa.dat
./dir1/dirb
./dir1/dirb/fileb.dat
./dir1/file1.dat
./dir1/file1.txt
./dir2
./dir2/emptydir
./dir2/file2.dat

Logical operations are used to group expressions. For example, the below command selects all files that have either a

.dat
 or
.txt
 extension using the OR logical operator.

$ find . -name "*.dat" -o -name "*.txt"
./dir1/dira/dirA/fileA.dat
./dir1/dira/filea.dat
./dir1/dira/fileaaa.dat
./dir1/dirb/fileb.dat
./dir1/file1.dat
./dir1/file1.txt
./dir2/file2.dat

Sometimes, however, you will need to use parenthesis (

(
  and
)
 ) to group expressions.  For example, the following command does not generate what you expect. Instead of printing all files with 
.dat
 or
.txt
 extensions, it just prints out the files with 
*.txt
 extension:

$ find . -name "*.dat" -o -name "*.txt" -print
./dir1/file1.txt

This command is equivalent to the command below. Remember that the logical AND (

-a
  ) is used by default if no explicit logical expression is provided:

$ find . -name "*.dat" -o \( -name "*.txt" -a -print \)

To print out all files matching  

.dat
 or
.txt
, you should group the two expressions inside a parenthesis. This command first correctly evaluates the condition, and then prints files matching that criteria:

$ find . \( -name "*.dat" -o -name "*.txt" \) -print

Of course, you could also use the NOT operator to exclude files that meet a condition:

$ find . -type f \! -name "*dat"

By realising that find is just a tool that evaluates expressions on all files, you can end up creating complex commands such as the below to execute different actions for different files:

$ find . \( -name "*.dat" -a -exec echo DAT: {} \; \) -o \
>        \( -name "*.txt" -a -exec echo TXT: {} \; \) -o \
>        \( -type d -a -exec echo DIR: {} \; \)

The above command makes use of a peculiar feature of the AND and OR logical expressions:

Expression 1 Operator Expression 2
TRUE -a (AND) Always evaluated
FALSE -a (AND) Never evaluated
TRUE -o (OR) Never evaluated
FALSE -o (OR) Always evaluated

In other words, if the first expression is FALSE, the AND operator does not need to evaluate the second expression because we already know the whole expression will be FALSE.  Similarly, when the first expression is FALSE using the OR operator, we have to evaluate the second expression as it could be TRUE.

Bibliography

The post Find command in Linux appeared first on Actimem.

]]>
http://actimem.com/linux/find/feed/ 0 598
Sed stream editor http://actimem.com/linux/sed/ http://actimem.com/linux/sed/#respond Mon, 29 Aug 2016 16:13:19 +0000 http://actimem.com/?p=536 This page will give you a brief introduction to sed and its main use cases.

The post Sed stream editor appeared first on Actimem.

]]>
Introduction

Sed stands for Stream EDitor. In traditional editors such as vi or emails, the user moves the cursor around the file and interactively update the data. Stream editors, on the other hand, apply a set of commands to a stream of data read from either the standard input or file.

This page will give you a brief introduction to sed and its main use cases.

How Sed Works

This is how sed processes data:

  1. Reads next line from the input stream, and copy it to the pattern space
  2. Executes all sed commands in order,
    • Some commands will make changes to the pattern space
    • Some commands will only execute when a condition is met
  3. Writes result of pattern space to the output stream
  4. Goes back to step 1 until all lines consumed

It should be noted that sed does not change the original line.

All changes happen in the pattern space and outputted to standard output.

Sed Command-Line Syntax

This is the syntax of sed:

sed [-n] [-e] 'commands' [inputfile]

sed [-n] -f scriptfile [inputfile]

Options:

-n Suppresses the output.

By default sed displays all lines to the standard output. When setting the

-n
  flag, sed only outputs the transformed line when when using the
p
  command, or the
p
  flag with the
s
  command.

sed -n '/second/p' lines.data
-e Adds a script to the commands to be executed. Useful when executing more than one command

sed -e 's/milk/mojitos/' -e 's/like/love/' test.data
commands one or several sed commands to execute
inputfile The input file to process. If not provided, then it will use the standard input
-f scriptfile File containing the sed commands to execute

Invoking Sed

This section contains several examples showing the basic usage of sed making use of the

s
  command (search and replace).

The first example reads the contents from the input stream and writes them to the output stream after replacing “milk” for “mojitos”.

$ echo 'I like drinking milk' | sed 's/milk/mojitos/'
I like drinking mojitos

You can alternatively provide an input file.

$ cat test.data
I like drinking milk in the morning
I like drinking milk in the morning
I like drinking milk in the morning

$ sed 's/milk/mojitos/' test.data
I like drinking mojitos in the morning
I like drinking mojitos in the morning
I like drinking mojitos in the morning

Sed can also execute multiple commands separated by semi-colon:

$ sed 's/milk/mojitos/; s/in the morning/on the beach/' test.data
I like drinking mojitos on the beach
I like drinking mojitos on the beach
I like drinking mojitos on the beach
I like drinking mojitos on the beach

Or you could also provide several scripts using the

-e
  flag:

$ sed -e 's/milk/mojitos/' -e 's/like/love/' test.data
I love drinking mojitos in the morning
I love drinking mojitos in the morning
I love drinking mojitos in the morning

Alternatively, you could also separate each command into a different line in the command prompt to make it more readable:

$ sed '
> s/milk/mojitos/
> s/in the morning/on the beach/
> s/like/love/' test.data
I love drinking mojitos on the beach
I love drinking mojitos on the beach
I love drinking mojitos on the beach

Finally, you could also store the sed script into a file:

$ cat script.sed
s/milk/mojitos/
s/in the morning/on the beach/
s/like/love/

$ sed -f script.sed test.data
I love drinking mojitos on the beach
I love drinking mojitos on the beach
I love drinking mojitos on the beach
I love drinking mojitos on the beach

Sed Commands and Addresses

These are the most commonly used commands in sed. For a full list of commands, check the sed manual.

Command Description
s Search and replace
y Translates characters. For example, translate all instances of lower case a to upper case A
d Deletes a line
= Displays the line number
p Prints the line result (useful when using
sed -n
  to control what is printed to standard output
n Skips the current line and go to the next line
N Appends the next line to the end of the pattern space. The new line is separated by a newline

A command can have an address and arguments.  If the command does not have any address, then it is always executed. If the command has an address, then it is only executed when the address condition is met (more on addresses in the next section).

[address[,address]][!]command [arguments]

You can also specify multiple commands for an address by separating them with a semi-colon

[address[,address]][!]'command1; command2'

You could also link multiple commands to an address by using curly brackets (make sure there are no spaces after the brackets):

[address[,address]][!]{
command1
command2
}

Addresses

The address determines whether the command is to be applied to the line.  A command with no address is applied to all lines.

[address[,address]][!]command [arguments]

An address can be a line number, a range of lines, the symbol

$
  to indicate the last line, or a regular expression (using
/slashes/
 ).  If the address is followed by the
!
 symbol, then it applies to all lines that do not match the address.

Let’s see a few examples

Address Example Description
no address sed ‘s/milk/mojito/’ test.data Changes all lines
1 sed ‘1s/milk/mojito/’ test.data Changes the first line only
$ sed ‘$s/milk/mojito/’ test.data Changes the last line
1,$ sed ‘1,$s/milk/mojito/’ test.data Changes all lines
2! sed ‘2!s/milk/mojito/’ test.data Changes all lines but number 2
/drinking/ sed ‘/drinking/s/milk/mojito/’ test.data Changes all lines that match the regular expression /drinking/
/start/,/end/! sed ‘/start/,/end/!s/milk/mojito/’ test.data Change all lines except the ones between the regular expressions /start/ and /end/

s command: Search and Replace

The 

s
 command is the most commonly used command in sed, used to search and replace within a file.  The
s
 command has a search and a replace section. The below  example replaces milk for mojitos.

$ echo 'I like drinking milk' | sed 's/milk/mojitos/'
I like drinking mojitos

The forward slash delimiter (

/
 ) is typically used to separate the search pattern from the replace part. We could however use alternative delimiters. This is particularly useful when working with path names.

$ echo 'I like drinking milk' | sed 's@milk@mojitos@'
I like drinking mojitos

The backwards slash (

\
) can also be used to match special characters such as
\s
,
\w
,
\n
,
\t
. The following example uses
\n
  to transform each word into a separate line:

$ echo "First Second Third Fourth" | sed 's/\s/\n/g'
First
Second
Third
Fourth

When using regular expressions, you can refer to the matched pattern using the ampersand (

&
).

echo "Chapter 12.3" | sed 's/[1-9][0-9]*/[&]/'
Chapter [12].3

You could also group multiple regular expressions enclosing them in parenthesis, and then reference them in the replace section using

\1
 ,
\2
 ,
\3
,…  Note that the parenthesis needs to be escaped (e.g.
\(
  and
\)
 )

$ echo "Chapter 12.3" | sed 's/\(Chapter\) \([1-9][0-9]*\)/[\1][\2]/'
[Chapter][12].3

By default, sed replaces only the first match:

$ echo "First Second Third Fourth" | sed 's/\w*/XXX/'
XXX Second Third Fourth

You can however use the

g
  flag to replace all matches in the line:

$ echo "First Second Third Fourth" | sed 's/\w*/XXX/g'
XXX XXX XXX XXX

You could alternatively specify what match number to use:

$ echo "First Second Third Fourth" | sed 's/\w*/XXX/2'
First XXX Third Fourth

$ echo "First Second Third Fourth" | sed 's/\w*/XXX/4'
First Second Third XXX

Note that sed does not support POSIX extended regular expressions (so you can’t use

+
 ,
?
  or
|
).

y command: Translate Characters

The

y
 command replaces any character in the search section by the character in the same position in the replace section.

The following example converts lowercase to uppercase:

$ echo "chapter number" | sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/'
CHAPTER NUMBER

d command: Delete Lines

The

d
 command deletes all lines that match the address.

For example, let’s assume we have the following file:

$ cat lines.data
This is the first line
This is the second line
This is the third line
This is the fourth line

The following command deletes the third line:

$ sed '3d' lines.data
This is the first line
This is the second line
This is the fourth line

The following command deletes all lines between pattern

/second/
 and
/third/
:

$ sed '/second/,/third/d' lines.data
This is the first line
This is the fourth line

A useful usage of the

d
 command is to delete empty lines in a file using the
/^$/
  address:

$ cat lines-with-spaces.data
This is the first line

This is the second line


This is the third line

$ sed '/^$/d' lines-with-spaces.data
This is the first line
This is the second line
This is the third line

p command: Print Lines

The

p
 command prints the pattern space to the standard output.

By default the line is always printed at the end of processing. The

p
  command is typically used in combination with sed’s
-n
 flag to selectively print lines to the standard output.

$ sed -n '/second/p' lines.data
This is the second line

If we didn’t use the

-n
 flag, then the second line would be printed twice:

$ sed '/second/p' lines.data
This is the first line
This is the second line
This is the second line
This is the third line
This is the fourth line

= command: Print Line numbers

The

=
 command prints out the line numbers.

$ sed '=' lines.data
1
This is the first line
2
This is the second line
3
This is the third line
4
This is the fourth line

If you want to output the line number in the same line as the contents, then you can use the following trick using the

N
 multi line command (see next section):

$ sed '=' lines.data | sed 'N; s/\n/ /'
1 This is the first line
2 This is the second line
3 This is the third line
4 This is the fourth line

You could also combine it with the

-n
 flag and the
p
 command to selectively print out lines.  The following example selects all lines between 
/second/
 and
/third/
 patterns, and prints out the line number.

$ sed -n '/second/,/third/{=;p}' lines.data
2
This is the second line
3
This is the third line

n and N commands: Read Next Line

The

n
 command reads the next line into the pattern space for processing. This command is useful for processing multiple lines at the same time.

For example, let’s assume we have a file with names and ages split in two lines, and each record separated by a dotted line:

$ cat contacts.data
Name: John Smith
Age: 23
-----------------
Name: Tom Bradley
Age: 44
-----------------
Name: Abraham Scott
Age: 33
-----------------

The following multi-line sed script searches for lines with the “Name” word, prints out the name, reads the following line (

n
 command), and finally prints out the age:

$ sed -n '/Name/{
> s/Name: \(.*\)$/\1/
> p
> n
> s/Age: \([0-9]*\)/\1 years old/
> p}' contacts.data

John Smith
23 years old
Tom Bradley
44 years old
Abraham Scott
33 years old

The

N
 command is slightly different to
n
 command, in that it appends the next line into the pattern space.  The first and newly appended line are separated by a new line.

$ sed -n '/Name/{
> N
> s/Name: \(.*\)\nAge: \([0-9]\)/\1 \2/
> p}' contacts.data
John Smith 23
Tom Bradley 44
Abraham Scott 33

Bibliography

The post Sed stream editor appeared first on Actimem.

]]>
http://actimem.com/linux/sed/feed/ 0 536
JAXB Annotations Reference http://actimem.com/java/jaxb-annotations/ http://actimem.com/java/jaxb-annotations/#respond Sun, 01 Nov 2015 11:30:07 +0000 http://actimem.com/?p=493 Introduction JAXB defines many annotations that shape how Java classes map to XML Schema. This article documents the JAXB annotations that you will use most often and provides practical examples how to use them. This reference page is part of a series of articles on JAXB: Introduction to JAXB JAXB Validation XSLT with JAXB Schemagen – convert Java … Continue reading JAXB Annotations Reference

The post JAXB Annotations Reference appeared first on Actimem.

]]>
Introduction

JAXB defines many annotations that shape how Java classes map to XML Schema. This article documents the JAXB annotations that you will use most often and provides practical examples how to use them.

This reference page is part of a series of articles on JAXB:

Index

These are the most common JAXB annotations:

Annotation Description
@XmlSchema Maps a Java package to an XML namespace.
@XmlAccessorType Defines the fields and properties of your Java classes that the JAXB engine uses for binding. It has four values: PUBLIC_MEMBER, FIELD, PROPERTY and NONE.
@XmlAccessOrder Defines the sequential order of the children.
@XmlType Maps a Java class to a schema type. It defines the type name and order of its children.
@XmlRootElement Defines the XML root element. Root Java classes need to be registered with the JAXB context when it is created.
@XmlElement Maps a field or property to an XML element
@XmlAttribute Maps a field or property to an XML attribute
@XmlTransient Prevents mapping a field or property to the XML Schema
@XmlValue Maps a field or property to the text value on an XML tag.
@XmlList Maps a collection to a list of values separated by space.
@XmlElementWrapper Maps a Java collection to an XML wrapped collection
@XmlJavaTypeAdapter Adapts any Java class to an XML Schema type

@XmlSchema

The @XmlSchema tag maps Java packages to XML namespaces. It is defined at package level, so the best place to declare it is in the package-info.java.

The @XmlSchema most common configuration is to provide the namespace and elementFormDefault attributes:

@XmlSchema(namespace = "http://actimem.com/xsd/company.xsd",
           elementFormDefault = XmlNsForm.QUALIFIED)
package com.actimem.blog.jaxb.namespace.defaultnamespace;

This @XmlSchema declaration generates an XML Schema like the one below:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" 
           xmlns="http://actimem.com/blog/jaxb/company.xsd"  
           targetNamespace="http://actimem.com/xsd/company.xsd" 
           elementFormDefault="qualified">

The namespace attribute in @XmlSchema maps to the targetNamespace attribute in xs:schema. The targetNamespace attribute has the namespace of the domain that the XML Schema defines.

The elementFormDefault attribute controls whether the elements must be namespace-qualified. Most XML Schema declarations set this to “qualified“. A qualified element means that all elements must belong to an XML Schema vocabulary. They must either have a namespace prefix or be part of the default namespace. There is also a counterpart attributeFormDefault, but is usually left as unqualified. For a good explanation of elementFormDefault and attributeFormDefault you can read Introduction to XML Schema.

Another usage of @XmlSchema is to declare multiple namespaces.

@XmlSchema(
    namespace = "http://actimem.com/xsd/company.xsd",
    elementFormDefault = XmlNsForm.QUALIFIED,
    xmlns = {
      @XmlNs(prefix="", namespaceURI="http://actimem.com/xsd/company.xsd"),
      @XmlNs(prefix="e", namespaceURI="http://actimem.com/xsd/employee.xsd")
    }
)
package com.actimem.blog.jaxb.namespace.multinamespace;

This declaration defines two namespaces, a default http://actimem.com/xsd/company.xsd and a prefixed http://actimem.com/xsd/employee.xsd. An example XML looks like this:

<company id="0" 
        xmlns="http://actimem.com/xsd/company.xsd"  
        xmlns:e="http://actimem.com/xsd/employee.xsd">
    <name>Actimem</name>
    <ranking>0</ranking>
    <employees>
        <e:employee>
            <e:age>25</e:age>
            <e:name>John</e:name>
        </e:employee>
        <e:employee>
            <e:age>32</e:age>
            <e:name>Brad</e:name>
        </e:employee>
    </employees>
</company>

@XmlAccessorType

The @XmlAccessorType defines the fields and properties of your Java classes that the JAXB engine uses for binding. It has four values: PUBLIC_MEMBER, FIELD, PROPERTY and NONE.

PUBLIC_MEMBER It binds public fields, annotated fields and properties (default)
FIELD It binds fields and annotated properties
PROPERTY It binds annotated fields and properties
NONE It binds annotated fields and annotated properties

PUBLIC_MEMBER

PUBLIC_MEMBER is the default accessor type. JAXB generates bindings for:

  • Public fields
  • Annotated fields
  • Properties

The following example has Java class with a PUBLIC_MEMBER accessor type.

@XmlRootElement
@XmlAccessorType(XmlAccessType.PUBLIC_MEMBER)
public class PublicMember {
    private String field;

    @XmlAttribute
    private String attribute;

    public String publicField;

    public String getFieldProperty() {
        return field;
    }
    public void setFieldProperty(String value) {
        this.field = value;
    }

    @XmlAttribute
    public String getAttributeProperty() {
        return attribute;
    }
    public void setAttributeProperty(String value) {
        this.attribute = value;
    }
}

When marshalled generates the following XML:

<publicMember attribute="attribute" attributeProperty="attribute">
    <publicField>public field</publicField>
    <fieldProperty>field</fieldProperty>
</publicMember>

FIELD

The FIELD accessor type creates bindings for:

  • Fields
  • Annotated properties

The following example has Java class with a FIELD accessor type.

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class Field {
    private String field;

    @XmlAttribute
    private String attribute;

    public String publicField;

    public String getFieldProperty() {
        return field;
    }
    public void setFieldProperty(String value) {
        this.field = value;
    }

    @XmlAttribute
    public String getAttributeProperty() {
        return attribute;
    }
    public void setAttributeProperty(String value) {
        this.attribute = value;
    }
}

When marshalled generates the following XML:

<field attribute="attribute" attributeProperty="attribute">
    <field>field</field>
    <publicField>public field</publicField>
</field>

PROPERTY

The PROPERTY accessor type creates bindings for:

  • Annotated fields
  • Properties

Property is very similar to PUBLIC_MEMBER. The only difference is that it ignores public fields. The benefit of using PROPERTY over FIELD is that some frameworks such as JPA might inject additional fields.

The following example has a Java class with a PROPERTY accessor type.

@XmlRootElement
@XmlAccessorType(XmlAccessType.PROPERTY)
public class Property {
    private String field;

    @XmlAttribute
    private String attribute;

    public String publicField;

    public String getFieldProperty() {
        return field;
    }
    public void setFieldProperty(String value) {
        this.field = value;
    }

    @XmlAttribute
    public String getAttributeProperty() {
        return attribute;
    }
    public void setAttributeProperty(String value) {
        this.attribute = value;
    }
}

When marshalled generates the following XML:

<property attribute="attribute" attributeProperty="attribute">
    <fieldProperty>field</fieldProperty>
</property>

NONE

The NONE accessor type creates bindings for:

  • Annotated fields
  • Annotated properties

This accessor type is useful when you want to have absolute control. JAXB only binds fields and properties annotated with @XmlAttribute or @XmlElement.

@XmlAccessOrder

The @XmlAccessOrder defines the sequential order of the children. It can have two values:

  • ALPHABETICAL
  • UNDEFINED (default)

When UNDEFINED is used, most JAXB providers use the same order as fields and properties are defined in the Java class. This behaviour is however not guaranteed. The only portable way to keep the order consistent across JAXB providers is:

  • Set @XmlAccessOrder to ALPHABETICAL
  • Define the order in the propOrder attribute of the @XmlType annotation

@XmlType

The @XmlType annotation maps a Java class to a schema type. The @XmlType defines the type name, namespace and order of its children.

The @XmlType annotation has two attributes: name and propOrder. The name attribute defines the name of the type used in the XML Schema. The propOrder defines the sequential order of the children. If propOrder is not present, then the order is defined by the @XmlAccessOrder annotation.

This example illustrates the usage of @XmlType:

@XmlRootElement(name="super-company")
@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name="companyType",
        propOrder = {"name", "foundedDate", 
                     "probabilityDefault", "ranking"})
public class Company {
    private String name;
    private int ranking;
    private double probabilityDefault;
    private Date foundedDate;
    ...
}

This is the XML Schema generated:

<xs:schema version="1.0" xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="super-company" type="companyType"/>

  <xs:complexType name="companyType">
    <xs:sequence>
      <xs:element name="name" type="xs:string" minOccurs="0"/>
      <xs:element name="founded" type="xs:dateTime" minOccurs="0"/>
      <xs:element name="default-prob" type="xs:double"/>
      <xs:element name="rank" type="xs:int"/>
    </xs:sequence>
    <xs:attribute name="id" type="xs:int" use="required"/>
  </xs:complexType>
</xs:schema>

This is the XML generated:

<super-company>
    <name>Actimem</name>
    <foundedDate>2003-03-03T00:00:00Z</foundedDate>
    <probabilityDefault>0.3</probabilityDefault>
    <ranking>10</ranking>
</super-company>

@XmlRootElement

The @XmlRootElement defines an XML root element. Root Java classes need to be registered when the JAXB context when it is created.

The following example defines a root class:

@XmlRootElement
public class Company {
    ...
}

You can also use the name attribute to change the root-node element name:

@XmlRootElement(name = "super-company")
public class Company {
    ...
}

You can marshal and unmarshal this Java class as follows:

JAXBContext ctx = JAXBContext.newInstance(Company.class);
            
StringWriter writer = new StringWriter();
ctx.createMarshaller().marshal(company, writer);
String xml = writer.toString();
        
Company unmarshalledCompany = (Company)ctx.createUnmarshaller().unmarshal(new StringReader(xml));

@XmlElement

The @XmlElement annotation maps a field or property to an XML element. By default, JAXB maps non-annotated Java field and properties to elements.

The @XmlElement annotation has four attributes of interest:

name The name of the element in the XML Schema
namespace The namespace of the element, if different to the parent node.
required Controls the minOccurs XML Schema attribute (default false)

If required=”true” then minOccurs=”1″. If required=”false” then maxOccurs=”0″

nillable Support for XML Schema nillable declarations (default false)

The following Java class illustrates the several usages of the @XmlElement annotation:

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class ElementExample {
    private String nonAnnotatedElement;

    @XmlElement
    private String nameDefault;

    @XmlElement(name = "nameChanged")
    private String anotherName;

    @XmlElement(required = true)
    private String required;

    @XmlElement(required = false)
    private String notRequired;

    @XmlElement(nillable = true)
    private String nillable;

    @XmlElement(nillable = false)
    private String notNillable;

    ...
}

The schemagen tool will generate the following XML Schema:

<xs:schema version="1.0" xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="elementExample" type="elementExample"/>

  <xs:complexType name="elementExample">
    <xs:sequence>
      <xs:element name="nonAnnotatedElement" type="xs:string" minOccurs="0"/>
      <xs:element name="nameDefault" type="xs:string" minOccurs="0"/>
      <xs:element name="nameChanged" type="xs:string" minOccurs="0"/>
      <xs:element name="required" type="xs:string"/>
      <xs:element name="notRequired" type="xs:string" minOccurs="0"/>
      <xs:element name="nillable" type="xs:string" nillable="true" minOccurs="0"/>
      <xs:element name="notNillable" type="xs:string" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>
</xs:schema>

The nillable attribute supports nillable XML elements. The code below marshals an ElementExample with all its non-required elements set to null.

ElementExample nillableExample = new ElementExample();
nillableExample.setRequired("required");

JAXBContext ctx = JAXBContext.newInstance(ElementExample.class);
Marshaller marshaller = ctx.createMarshaller();
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
marshaller.marshal(elementExample, System.out);

marshaller.marshal(nillableExample, System.out);

The XML nillable element is represented with the XMLSchema-instance xsi:nil=”true”

<elementExample>
    <required>required</required>
    <nillable xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
</elementExample>

@XmlAttribute

The @XmlAttribute maps a field or property to an XML attribute. It has three attributes of interest:

name The name of the element in the XML Schema
namespace The namespace of the element, if different to the parent node.
required Controls the minOccurs XML Schema attribute (default false)

If required=”true” then minOccurs=”1″. If required=”false” then maxOccurs=”0″

The following Java class illustrate the usage of @XmlAttribute:

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class AttributeExample {
    @XmlAttribute
    private String defaultAttribute;

    @XmlAttribute(name = "named-attribute")
    private String namedAttribute;

    @XmlAttribute(required = true)
    private String requiredAttribute;

    @XmlAttribute(required = false)
    private String optionalAttribute;
    ...
}

This class generates the following XML Schema:

<xs:schema version="1.0" xmlns:xs="http://www.w3.org/2001/XMLSchema">
    
  <xs:element name="attributeExample" type="attributeExample"/>

  <xs:complexType name="attributeExample">
    <xs:sequence/>
    <xs:attribute name="defaultAttribute" type="xs:string"/>
    <xs:attribute name="named-attribute" type="xs:string"/>
    <xs:attribute name="requiredAttribute" type="xs:string" use="required"/>
    <xs:attribute name="optionalAttribute" type="xs:string"/>
  </xs:complexType>
</xs:schema>

The @XmlAttribute can also annotate Lists. The only constraint is that the values in the list must map to a simple XML Schema type (e.g. xs:string, xs:int, xs:date, etc.). The following Java class illustrates this:

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class AttributeListExample {
    @XmlAttribute
    private List<String> listAttributes = new ArrayList<>();

    public List<String> getListAttributes() {
        return listAttributes;
    }

    public void setListAttributes(List<String> listAttributes) {
        this.listAttributes = listAttributes;
    }
}

The below code marshals the above Java class into XML:

public class AttributeDemo {
    public static void main(String[] args) throws JAXBException {
        JAXBContext context = JAXBContext.newInstance(AttributeExample.class);
        Marshaller marshaller = context.createMarshaller();
        marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
    
        AttributeListExample example = new AttributeListExample();
        example.getListAttributes().add("First");
        example.getListAttributes().add("Second");
        example.getListAttributes().add("Third");

        marshaller.marshal(example, System.out);
    }
}

It generates the following XML:

<attributeExample listAttributes="First Second Third"/>

@XmlTransient

The @XmlTransient annotation prevents mapping a field or property to the XML Schema.

JAXB maps all private fields and properties to an XML element when @XmlAccessorType is set to PUBLIC_MEMBER, PROPERTY or FIELD.  The @XmlTransient ensures the annotated field or property is not marshalled to XML.

If you have many fields or properties that you do not want to map, then you might consider setting @XmlAccessorType to NONE. When NONE, only the fields annotated with @XmlAnnotation or @XmlElement are mapped.

@XmlValue

The @XmlValue annotation maps a field or property to the text value of an XML tag. A Java class containing a field or property is constrained as follows:

  • It cannot have any field or properties annotated with @XmlElement
  • The field or property annotated with @XmlValue must map to a simple type

This Java class shows how to use the @XmlValue annotation:

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class ValueExample {
    @XmlAttribute
    private String name;

    @XmlValue
    private String value;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getValue() {
        return value;
    }

    public void setValue(String value) {
        this.value = value;
    }
}

When marshalled, it generates an XML similar to the below:

<valueExample name="name">A very large value</valueExample>

@XmlList

The @XmlList annotation maps a collection to a list of values separated by space. The only limitation is that the items in the collection must map to simple XML Schema types. This is to say, @XmlList does not support complex types.

The Java class below shows an example how to use the @XmlList annotation (the @XmlElement is optional):

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class XmlListExample {
    @XmlElement
    @XmlList
    private List<String> names = new ArrayList<>();

    public List<String> getNames() {
        return names;
    }
}

When adding three items to the names list, it generates the following XML:

<xmlListExample>
    <names>First Second Third</names>
</xmlListExample>

This contrasts with a Java collection that only annotated with @XmlElement.  When this happens, JAXB wraps each item in the collection with an element. So the below Java class:

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class XmlListExampleWrapped {
    private List<String> names = new ArrayList<>();

    public List<String> getNames() {
        return names;
    }
}

When marshalled, it generates this XML:

<xmlListExampleWrapped>
    <names>First</names>
    <names>Second</names>
    <names>Third</names>
</xmlListExampleWrapped>

@XmlElementWrapper

The @XmlElementWrapper maps a Java collection to an XML wrapped collection. JAXB supports two types of collection mappings, wrapped and unwrapped. The default is unwrapped.

The Java class below annotates the employees collection with the @XmlElementWrapper annotation.

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class Company {
    private String name;

    @XmlElementWrapper(name="employees")
    @XmlElement(name="employee")
    private List<Employee> employees = new ArrayList<>();

    public String getName() { return name; }
    public void setName(String name) { this.name = name; }

    public List<Employee> getEmployees() { return employees; }
}

This is the XML generated. As you can see, the items in the list are wrapped by the employees XML element.

<company>
    <name>Actimem</name>
    <employees>
        <employee>
            <age>25</age>
            <name>John</name>
        </employee>
        <employee>
            <age>32</age>
            <name>Brad</name>
        </employee>
    </employees>
</company>

If we didn’t use the @XmlElementWrapper annotation, this is the XML that would be generated. Note that each item in the list is a direct child of company.

<company>
    <name>Actimem</name>
    <employees>
        <age>25</age>
        <name>John</name>
    </employees>
    <employees>
        <age>32</age>
        <name>Brad</name>
    </employees>
</company>

@XmlJavaTypeAdapter and @XmlJavaTypeAdapters

The @XmlJavaTypeAdapter is the workhorse of JAXB type mapping. JAXB can map any Java class to an XML Schema type using the @XmlJavaTypeAdapter annotation.

The following class illustrates how to use adapters. The Company class has three elements that map to non-standard XML Schema types: CompanyId, and Joda’s DateTime and LocalDate.

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class Company {
    private CompanyId companyId;
    private String name;
    private LocalDate foundedDate;
    private DateTime createdAt;

    public CompanyId getCompanyId() { return companyId; }
    public void setCompanyId(CompanyId companyId) { this.companyId = companyId; }

    public DateTime getCreatedAt() { return createdAt; }
    public void setCreatedAt(DateTime createdAt) { this.createdAt = createdAt; }

    public LocalDate getFoundedDate() { return foundedDate; }
    public void setFoundedDate(LocalDate foundedDate) { this.foundedDate = foundedDate; }

    public String getName() { return name; }
    public void setName(String name) { this.name = name; }
}

This is the implementation of the @XmlJavaTypeAdapter to support Joda DateTimes. The adapter extends from javax.xml.bind.annotation.adapters.XmlAdapter and must implement the marshal and umarshal methods.

public class DateTimeAdapter extends XmlAdapter<String, DateTime> {
    @Override
    public DateTime unmarshal(String v) throws Exception {
        return new DateTime(v);
    }

    @Override
    public String marshal(DateTime v) throws Exception {
        return v.toString();
    }
}

Finally, we need to register the adapters for the package in the package-info.java file so that the JAXB engine can use them:

@XmlJavaTypeAdapters({
        @XmlJavaTypeAdapter(type=DateTime.class, value = DateTimeAdapter.class),
        @XmlJavaTypeAdapter(type=LocalDate.class, value = LocalDateAdapter.class),
        @XmlJavaTypeAdapter(type=CompanyId.class, value = CompanyIdAdapter.class)
})
package com.actimem.blog.jaxb.adapters;

Note that you could also declare the @XmlJavaTypeAdapter annotation at method or class level.

Bibliography

The post JAXB Annotations Reference appeared first on Actimem.

]]>
http://actimem.com/java/jaxb-annotations/feed/ 0 493
xjc – Convert XML Schema to Java http://actimem.com/java/xjc/ http://actimem.com/java/xjc/#respond Sun, 01 Nov 2015 10:47:47 +0000 http://actimem.com/?p=488 Introduction The xjc tool generates Java source code from an XML Schema. This article is a quick reference page to use xjc from both the command line and Maven. This reference page is part of a series of articles on JAXB: Introduction to JAXB JAXB Validation XSLT with JAXB Schemagen – convert Java classes to XML Schema Xjc – convert Xml … Continue reading xjc – Convert XML Schema to Java

The post xjc – Convert XML Schema to Java appeared first on Actimem.

]]>
Introduction

The xjc tool generates Java source code from an XML Schema. This article is a quick reference page to use xjc from both the command line and Maven.

This reference page is part of a series of articles on JAXB:

Command Line Syntax

The xjc syntax below generates the Java source code from the company.xsd file.

xjc -d src company.xsd

You need to ensure the src directory exists, as otherwise nothing will happen.

And this is it! For more details check the official documentation.

Maven

If you use maven, you can use the JAXB-2 Maven Plugin to generate Java source. This is how you declare the plugin in the pom.xml file:

<build>
  <plugins>
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>jaxb2-maven-plugin</artifactId>
      <version>2.2</version>
      <executions>
        <execution>
          <id>xjc</id>
          <goals>
            <goal>xjc</goal>
          </goals>
          <configuration>
            <packageName>com.actimem.blog.jaxb.xjc.sources</packageName>
            <noGeneratedHeaderComments>true</noGeneratedHeaderComments>
            <sources>
              <source>src/main/resources/examples/xjc</source>
            </sources>
          </configuration>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

The source parameter above determines the location of the .xsd file. It will process all .xsd files in the src/main/resources/examples/xjc directory. If you have .xsd files in multiple locations, then just add another source entry. The packageName parameter determines the package of the sources generated. It overrides any package level annotations declared in the .xsd files.

Bibliography

The post xjc – Convert XML Schema to Java appeared first on Actimem.

]]>
http://actimem.com/java/xjc/feed/ 0 488
schemagen – Convert Java classes to XML Schema http://actimem.com/java/schemagen/ http://actimem.com/java/schemagen/#comments Sun, 01 Nov 2015 10:35:43 +0000 http://actimem.com/?p=482 Introduction The schemagen tool converts Java classes to a XML Schema. This article is a quick reference page to use schemagen from both the command line and Maven. This reference page is part of a series of articles on JAXB: Introduction to JAXB JAXB Validation XSLT with JAXB Schemagen – convert Java classes to XML Schema Xjc – convert Xml Schema … Continue reading schemagen – Convert Java classes to XML Schema

The post schemagen – Convert Java classes to XML Schema appeared first on Actimem.

]]>
Introduction

The schemagen tool converts Java classes to a XML Schema. This article is a quick reference page to use schemagen from both the command line and Maven.

This reference page is part of a series of articles on JAXB:

Command Line Syntax

The schemagen syntax below generates an .xsd XML Schema file for the Company class and dependants.

schemagen -cp . com.actimem.blog.jaxb.schemagen.Company

You could also use the -d option to define the location where the .xsd file will be created.

schemagen -d xsd_files -cp . com.actimem.blog.jaxb.schemagen.Company

The schemagen tool can generate a XML Schema from either .java or .class files. If you generate it from .java files, you need to be aware that it will compile and leave the .class files in the source directory. This is the syntax:

schemagen -cp . com/actimem/blog/jaxb/schemagen/Company.java

For more details on the schemagen syntax check the official documentation.

Maven Configuration

If you use maven, you can use the JAXB-2 Maven Plugin to generate the XML Schema files. This is how you declare the plugin in the pom.xml file:

<build>
  <plugins>
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>jaxb2-maven-plugin</artifactId>
      <version>2.2</version>
      <executions>
        <execution>
          <id>schemagen</id>
          <goals>
            <goal>schemagen</goal>
          </goals>
          <configuration>
            <sources>
             <source>src/main/java/com/actimem/blog/jaxb/schemagen</source>
            </sources>
            <transformSchemas>
              <transformSchema>
                <uri>http://actimem.com/blog/jaxb/company</uri>
                <toPrefix>c</toPrefix>
                <toFile>company.xsd</toFile>
              </transformSchema>
            </transformSchemas>
          </configuration>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

The above configuration generates an XML Schema for all classes in the com.actimem.blog.jaxb.schemagen package.  It also defines that all classes for namespace http://actimem.com/blog/jaxb/company are prefixed with letter “c“.  The company.xsd file will be generated in the target/generated-resources/schemagen directory.

Bibliography

The post schemagen – Convert Java classes to XML Schema appeared first on Actimem.

]]>
http://actimem.com/java/schemagen/feed/ 1 482
XSLT with JAXB http://actimem.com/java/xslt-with-jaxb/ http://actimem.com/java/xslt-with-jaxb/#respond Sun, 01 Nov 2015 10:14:10 +0000 http://actimem.com/?p=478 Introduction This tutorial explains how to transform Java classes using XSLT templates. XSLT is a language that transforms XML documents to another format, such as HTML, PDF or XML with different structure. JAXB lets you treat your Java objects as if they were XML documents. We can therefore leverage the existing javax.xml.transform APIs to transform … Continue reading XSLT with JAXB

The post XSLT with JAXB appeared first on Actimem.

]]>
Introduction

This tutorial explains how to transform Java classes using XSLT templates. XSLT is a language that transforms XML documents to another format, such as HTML, PDF or XML with different structure. JAXB lets you treat your Java objects as if they were XML documents. We can therefore leverage the existing javax.xml.transform APIs to transform Java objects using XSLT.

This tutorial is part of a series of articles on JAXB:

The Domain Model

This tutorial uses a simple JAXB Java domain. A Company has a name and a list of Employees.

This is the Company class:

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class Company {
    private String name;

    @XmlElementWrapper(name="employees")
    @XmlElement(name="employee")
    private List<Employee> employees = new ArrayList<>();

    public String getName() { return name; }
    public void setName(String name) { this.name = name; }

    public List<Employee> getEmployees() { return employees; }
}

This is the Employee class:

public class Employee {
    private String name;
    private int age;

    public int getAge() { return age; }
    public void setAge(int age) { this.age = age; }

    public String getName() { return name; }
    public void setName(String name) { this.name = name; }
}

The XSLT Template

This XSLT template transforms our Java classes into HTML.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:template match="/">
        <html>
            <body>
                <h2>Company Information</h2>
                <p>
                    <b>Name:</b> <xsl:value-of select="company/name"/>
                </p>
                <b>Employees</b>
                <table>
                    <xsl:for-each select="company/employees/employee">
                        <tr>
                            <td><xsl:value-of select="name"/></td>
                            <td><xsl:value-of select="age"/></td>
                        </tr>
                    </xsl:for-each>
                </table>
            </body>
        </html>
    </xsl:template>
</xsl:stylesheet>

Don’t worry if you don’t know much about XSLT. This tutorial only focuses on how to transform using XSLT rather than creating XSLT templates. If you want to know more about XSLT, you can start with this XSLT Tutorial.

Transforming JAXB to HTML

The XsltDemo below has three steps:

  • Creates a Company with two Employees
  • Creates a javax.xml.transform.Transformer instance based on the company.xsl XSLT template
  • Transforms a JAXB Java class into HTML

public class XsltDemo {
    public static void main(String[] args) throws JAXBException, TransformerException, IOException {
        Company company = new Company();
        company.setName("Actimem");
        Employee john = new Employee();
        john.setName("John");
        john.setAge(25);
        Employee brad = new Employee();
        brad.setName("Brad");
        brad.setAge(32);
        company.getEmployees().add(john);
        company.getEmployees().add(brad);

        TransformerFactory factory = TransformerFactory.newInstance();
        InputStream resourceAsStream = XsltDemo.class.getResourceAsStream("/examples/xslt/company.xsl");
        StreamSource xslt = new StreamSource(resourceAsStream);
        Transformer transformer = factory.newTransformer(xslt);

        JAXBContext context = JAXBContext.newInstance(Company.class);
        JAXBSource source = new JAXBSource(context, company);
        StreamResult result = new StreamResult(System.out);
        transformer.transform(source, result);
    }
}

The Transformer class transforms from an XML Source to an XML Result. JAXBSource implements the Source interface. This enables the XSLT Transformer to use JAXB objects as if they were XML documents.

This is the generated HTML:

<html>
<body>
<h2>Company Information</h2>
<p>
<b>Name:</b>Actimem</p>
<b>Employees</b>
<table>
<tr><td>John</td><td>25</td></tr>
<tr><td>Brad</td><td>32</td></tr>
</table>
</body>
</html>

Bibliography

The post XSLT with JAXB appeared first on Actimem.

]]>
http://actimem.com/java/xslt-with-jaxb/feed/ 0 478
JAXB Validation http://actimem.com/java/jaxb-validation/ http://actimem.com/java/jaxb-validation/#respond Sat, 31 Oct 2015 17:51:13 +0000 http://actimem.com/?p=475 Introduction This tutorial explains how to use JAXB validation. It first shows how to validate a JAXB object marshalled to XML. It then shows how to validate an XML message unmarshalled to a Java object. It finally explains how to leverage the javax.xml.validation API to validate a Java object against an XML Schema. This tutorial is part … Continue reading JAXB Validation

The post JAXB Validation appeared first on Actimem.

]]>
Introduction

This tutorial explains how to use JAXB validation. It first shows how to validate a JAXB object marshalled to XML. It then shows how to validate an XML message unmarshalled to a Java object. It finally explains how to leverage the javax.xml.validation API to validate a Java object against an XML Schema.

This tutorial is part of a series of articles on JAXB:

The Domain Model

All the examples in this section use the following XML Schema for validation (company.xsd):

<xs:schema version="1.0" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="company" type="company"/>
  <xs:complexType name="company">
    <xs:sequence>
      <xs:element name="name" type="xs:string" minOccurs="0"/>
      <xs:element name="countries" minOccurs="0">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="countries" type="xs:string" minOccurs="0" maxOccurs="3"/>
          </xs:sequence>
        </xs:complexType>
      </xs:element>
    </xs:sequence>
  </xs:complexType>
</xs:schema>

The Company entity has a name and a list of countries. The company.xsd, however, sets the restriction that a company can only have a maximum of three countries.

This is the Java annotated class:

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
public class Company {
    private String name;
    @XmlElementWrapper
    private List<String> countries = new ArrayList<>();

    public String getName() { return name; }
    public void setName(String name) { this.name = name; }

    public List<String> getCountries() { return countries; }
}

Marshalling Validation

The javax.xml.bind.Marshaller interface has two methods that enable validation:

marshaller.setSchema(schema);
marshaller.setEventHandler(companyErrorHandler);

The setSchema() method takes as parameter a javax.xml.validation.Schema. The Schema class is an XML Schema that can be used for validation.

SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
URL schemaURL = MarshallingDemo.class.getResource("/examples/validation/company.xsd");
Schema schema = schemaFactory.newSchema(schemaURL);

The setEventHandler() method takes a ValidationEventHandler that will handle all validation errors.  If the handleEvent() method returns true, the marshaller will continue with the processing. If it returns false, it will abort it.

private static ValidationEventHandler companyErrorHandler = new ValidationEventHandler() {
    @Override
    public boolean handleEvent(ValidationEvent event) {
        System.err.println(String.format("ERROR: %s (%d, %d) Severity: %s", event.getMessage(),
                event.getLocator().getLineNumber(),
                event.getLocator().getColumnNumber(),
                event.getSeverity()));
        return false;
    }
};

This is the complete MashallingDemo:

public class MarshallingDemo {
    public static void main(String[] args) throws Exception {
        Company company = new Company();
        company.setName("Actimem");
        company.getCountries().add("UK");
        company.getCountries().add("FR");
        company.getCountries().add("GE");
        company.getCountries().add("XX");

        SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
        URL schemaURL = MarshallingDemo.class.getResource("/examples/validation/company.xsd");
        Schema schema = schemaFactory.newSchema(schemaURL);

        JAXBContext context = JAXBContext.newInstance(Company.class);
        Marshaller marshaller = context.createMarshaller();
        marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
        marshaller.setSchema(schema);
        marshaller.setEventHandler(companyErrorHandler);
        marshaller.marshal(company, System.out);
    }

    private static ValidationEventHandler companyErrorHandler = new ValidationEventHandler() {
        @Override
        public boolean handleEvent(ValidationEvent event) {
            System.err.println(String.format("ERROR: %s (%d, %d) Severity: %s", event.getMessage(),
                    event.getLocator().getLineNumber(),
                    event.getLocator().getColumnNumber(),
                    event.getSeverity()));
            return false;
        }
    };
}

When executed, it will print the following validation error to the console, as the company has four countries, one more than allowed.

ERROR: cvc-complex-type.2.4.d: Invalid content was found starting with element 'countries'. No child element is expected at this point. (8, 20) Severity: 2

Unmarshalling Validation

Unmarshalling validation is very similar to Marshalling validation. The javax.xml.bind.Unmarshaller interface also has a setSchema() and setEventHandler() methods that work the same way.

This is the XML that will be unmarshalled. It will fail validation as it has four countries.

<company>
    <name>Actimem</name>
    <countries>
        <countries>UK</countries>
        <countries>FR</countries>
        <countries>GE</countries>
        <countries>XX</countries>
    </countries>
</company>

And this is the UnmarshallingDemo:

public class UnmarshallingDemo {
    public static void main(String[] args) throws Exception {
        URL schemaURL = UnmarshallingDemo.class.getResource("/examples/validation/company.xsd");
        URL xmlUrl = UnmarshallingDemo.class.getResource("/examples/validation/company.xml");

        SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
        Schema schema = schemaFactory.newSchema(schemaURL);

        JAXBContext context = JAXBContext.newInstance(Company.class);
        Unmarshaller unmarshaller = context.createUnmarshaller();
        unmarshaller.setSchema(schema);
        unmarshaller.setEventHandler(companyErrorHandler);
        unmarshaller.unmarshal(xmlUrl);
    }

    private static ValidationEventHandler companyErrorHandler = new ValidationEventHandler() {
        @Override
        public boolean handleEvent(ValidationEvent event) {
            System.err.println(String.format("ERROR: %s (%d, %d) Severity: %s", event.getMessage(),
                    event.getLocator().getLineNumber(),
                    event.getLocator().getColumnNumber(),
                    event.getSeverity()));
            return true;
        }
    };
}

When executed, it generates the same error as the marshalling example:

ERROR: cvc-complex-type.2.4.d: Invalid content was found starting with element 'countries'. No child element is expected at this point. (8, 20) Severity: 2

Validating JAXB with javax.xml.validation.Validator

JAXB can use the javax.xml.validation APIs to check the validity of Java classes against an XML Schema.

The validation takes place in a javax.xml.validation.Validator object. Validators are created from a javax.xml.validation.Schema object, which is the representation of an XML Schema. A validator accepts multiple XML inputs, represented by the javax.xml.transform.Source interface. Java provides several implementations such as DOMSource, JAXBSource, SAXSource, StAXSource and StreamSource.  To validate a JAXB Java object, you need to use JAXBSource.

It is also worth noting that validation errors are raised into an org.xml.sax.ErrorHandler.

This is the ValidationDemo:

public class ValidationDemo {
    public static void main(String[] args) throws Exception {
        Company company = new Company();
        company.setName("Actimem");
        company.getCountries().add("UK");
        company.getCountries().add("FR");
        company.getCountries().add("GE");
        company.getCountries().add("XX");

        JAXBContext context = JAXBContext.newInstance(Company.class);
        JAXBSource source = new JAXBSource(context, company);

        SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
        URL schemaURL = ValidationDemo.class.getResource("/examples/validation/company.xsd");
        Schema schema = schemaFactory.newSchema(schemaURL);

        Validator validator = schema.newValidator();
        validator.setErrorHandler(companyErrorHandler);
        validator.validate(source);
    }

    public static final ErrorHandler companyErrorHandler = new ErrorHandler() {
        @Override
        public void warning(SAXParseException exception) throws SAXException {
        }

        @Override
        public void error(SAXParseException exception) throws SAXException {
            System.err.println(String.format("ERROR: %s", exception));
        }

        @Override
        public void fatalError(SAXParseException exception) throws SAXException {
        }
    };
}

Bibliography

The post JAXB Validation appeared first on Actimem.

]]>
http://actimem.com/java/jaxb-validation/feed/ 0 475