I have bunch of keys (clientKey) and values (processBytes) that I want to send to our messaging queue by packing them in one byte array. I will make one byte array of all the keys and values which should always be less than 50K and then send to our messaging queue.
For each partition, I have bunch of dataHolders so I am iterating those and then sending it to my messaging queue:-
private void validateAndSend(final DataPartition partition) {
final ConcurrentLinkedQueue<DataHolder> dataHolders = dataHoldersByPartition.get(partition);
// sending data via async policy but it can be send with other two sync queue policy as well.
final Packet packet = new Packet(partition, new QPolicyAsync());
DataHolder dataHolder;
while ((dataHolder = dataHolders.poll()) != null) {
packet.addAndSendJunked(dataHolder.getClientKey().getBytes(StandardCharsets.UTF_8),
dataHolder.getProcessBytes());
}
packet.close();
}
Packet class: This class packs all the keys and values into one byte array and call corresponding implementation passed in the constructor to send data to queue.
public final class Packet implements Closeable {
private static final int MAX_SIZE = 50000;
private static final int HEADER_SIZE = 36;
private final byte dataCenter;
private final byte recordVersion;
private final long address;
private final long addressFrom;
private final long addressOrigin;
private final byte partition;
private final byte replicated;
private final ByteBuffer itemBuffer = ByteBuffer.allocate(MAX_SIZE);
private final QueuePolicy policy;
private int pendingItems = 0;
public Packet(final DataPartition partition, final QueuePolicy policy) {
this.partition = (byte) partition.getPartition();
this.policy = policy;
this.dataCenter = Utils.LOCATION.get().datacenter();
this.recordVersion = 1;
this.replicated = 0;
final long packedAddress = new Data().packAddress();
this.address = packedAddress;
this.addressFrom = 0L;
this.addressOrigin = packedAddress;
}
private void addHeader(final ByteBuffer buffer, final int items) {
buffer.put(dataCenter).put(recordVersion).putInt(items).putInt(buffer.capacity())
.putLong(address).putLong(addressFrom).putLong(addressOrigin).put(partition)
.put(replicated);
}
// sending here by calling policy implementation
private void sendData() {
if (itemBuffer.position() == 0) {
// no data to be sent
return;
}
final ByteBuffer buffer = ByteBuffer.allocate(MAX_SIZE);
addHeader(buffer, pendingItems);
buffer.put(itemBuffer);
// sending data via particular policy
policy.sendToQueue(address, buffer.array());
itemBuffer.clear();
pendingItems = 0;
}
public void addAndSendJunked(final byte[] key, final byte[] data) {
if (key.length > 255) {
return;
}
final byte keyLength = (byte) key.length;
final byte dataLength = (byte) data.length;
final int additionalSize = dataLength + keyLength + 1 + 1 + 8 + 2;
final int newSize = itemBuffer.position() + additionalSize;
if (newSize >= (MAX_SIZE - HEADER_SIZE)) {
sendData();
}
if (additionalSize > (MAX_SIZE - HEADER_SIZE)) {
throw new AppConfigurationException("Size of single item exceeds maximum size");
}
final ByteBuffer dataBuffer = ByteBuffer.wrap(data);
final long timestamp = dataLength > 10 ? dataBuffer.getLong(2) : System.currentTimeMillis();
// data layout
itemBuffer.put((byte) 0).put(keyLength).put(key).putLong(timestamp).putShort(dataLength)
.put(data);
pendingItems++;
}
@Override
public void close() {
if (pendingItems > 0) {
sendData();
}
}
}
Now I can send data to my messaging queue in three different ways so for that I created an interface and then having three different implementations:
QueuePolicy interface:
public interface QueuePolicy {
public boolean sendToQueue(final long address, final byte[] encodedRecords);
}
QPolicyAsync class:
public class QPolicyAsync implements QueuePolicy {
@Override
public boolean sendToQueue(long address, byte[] encodedRecords) {
return SendRecord.getInstance().sendToQueueAsync(address, encodedRecords);
}
}
QPolicySync class:
public class QPolicySync implements QueuePolicy {
@Override
public boolean sendToQueue(long address, byte[] encodedRecords) {
return SendRecord.getInstance().sendToQueueSync(address, encodedRecords);
}
}
QPolicySyncWithSocket class:
public class QPolicySyncWithSocket implements QueuePolicy {
private final Socket socket;
public QPolicySyncWithSocket(Socket socket) {
this.socket = socket;
}
@Override
public boolean sendToQueue(long address, byte[] encodedRecords) {
return SendRecord.getInstance().sendToQueueSync(address, encodedRecords, Optional.of(socket));
}
}
The idea is very simple: I am sending data to my messaging queue via either of those three QueuePolicy implementations. It depends on how clients want to send data. As of now I am passing implementation of QueuePolicy in the Packet constructor and then sends data via that policy. Each QueuePolicy implementation calls corresponding method in SendRecord class.
Now I need to know whether data was successfully sent or not. As of now methods in Packet class doesn't return any boolean so I don't know whether it was successfully sent or not. I can have cases where dataHolders has only one element in it or it can have multiple elements in it.
private void validateAndSend(final DataPartition partition) {
final ConcurrentLinkedQueue<DataHolder> dataHolders = dataHoldersByPartition.get(partition);
// sending data via async policy but it can be send with other two sync queue policy as well.
final Packet packet = new Packet(partition, new QPolicyAsync());
DataHolder dataHolder;
while ((dataHolder = dataHolders.poll()) != null) {
packet.addAndSendJunked(dataHolder.getClientKey().getBytes(StandardCharsets.UTF_8),
dataHolder.getProcessBytes());
}
packet.close();
// how do I know whether this data was successfully sent?
}
If I return boolean from addAndSendJunked and close method in Packet class, then which boolean value I need to rely on? Because either of these two methods can send the data.
closemethod will send data either there is only one element in it or there was left over elements.addAndSendJunkedmethod will send data as soon as limit is reached.
Aucun commentaire:
Enregistrer un commentaire