lundi 1 juin 2015

Pattern for building a functional based plugin architecture

I am investigating how to develop a plugin framework for a project and Rx seems like a good fit for what i am trying to achieve. Ultimately, the project will be a set of plugins (modular functionality) that can be configured via xml to do different things. The requirements are as follows

  1. Enforce a modular architecture even within a plugin. This encourages loose coupling and potentially minimizes complexity. This hopefully should make individual plugin functionality easier to model and test
  2. Enforce immutability with respect to data to reduce complexity and ensure that state management within modules is kept to a minimum
  3. Discourage manual thread creation by providing thread pool threads to do work within modules wherever possible

In my mind, a plugin is essentially a data transformation entity (I'm trying to think functional here). This means a plugin either

  • Takes in some data and transforms it in some way to produce new data (Not shown here)
  • Generates data in itself and pushes it out to observers
  • Takes in some data and does some work on the data without notifying outsiders

If you take the concept further, a plugin can consist of a number of all three types above.For example within a plugin you can have an IntGenerator module that generates some data to a ConsoleWorkUnit module etc. So what I am trying to model in the main function is the wiring that a plugin would have to do its work.

To that end, I have the following base classes using the Immutable nuget from Microsoft. What I am trying to achieve is to abstract away the Rx calls so they can be used in modules so the ultimate aim would be to wrap up calls to buffer etc in abstract classes that can be used to compose complex queries and modules. This way the code is a bit more self documenting than having to actually read all the code within a module to find out it subscribes to a buffer or window of type x etc.

public abstract class OutputBase<TOutput> : SendOutputBase<TOutput>
{
    public abstract void Work();
}

public interface IBufferedBase<TOutput>
{
    void Work(IList<ImmutableList<Data<TOutput>>> list);
}

public abstract class BufferedWorkBase<TInput> : IBufferedBase<TInput>
{
    public abstract void Work(IList<ImmutableList<Data<TInput>>> input);
}
public abstract class SendOutputBase<TOutput>
{
    private readonly ReplaySubject<ImmutableList<Data<TOutput>>> _outputNotifier;
    private readonly IObservable<ImmutableList<Data<TOutput>>> _observable;

    protected SendOutputBase()
    {
        _outputNotifier = new ReplaySubject<ImmutableList<Data<TOutput>>>(10);
        _observable  =  _outputNotifier.SubscribeOn(ThreadPoolScheduler.Instance);
        _observable = _outputNotifier.ObserveOn(ThreadPoolScheduler.Instance);
    }

    protected void SetOutputTo(ImmutableList<Data<TOutput>> output)
    {
        _outputNotifier.OnNext(output);
    }

    public void ConnectOutputTo(IWorkBase<TOutput> unit)
    {
        _observable.Subscribe(unit.Work);
    }

    public void BufferOutputTo(int count, IBufferedBase<TOutput> unit)
    {
        _observable.Buffer(count).Subscribe(unit.Work);
    }
}

public abstract class WorkBase<TInput> : IWorkBase<TInput>
{
    public abstract void Work(ImmutableList<Data<TInput>> input);
}

public interface IWorkBase<TInput>
{
    void Work(ImmutableList<Data<TInput>> input);
}

public class Data<T>
{
    private readonly T _value;

    private Data(T value)
    {
        _value = value;
    }

    public static Data<TData> Create<TData>(TData value)
    {
        return new Data<TData>(value);
    }

    public T Value { get { return _value; } }

}

These base classes are used to create three classes; one for generating some int data, one to print out the data when they occur and the last to buffer the data as it comes in and sum the values in threes.

public class IntGenerator : OutputBase<int>
{
    public override void Work()
    {
        var list = ImmutableList<Data<int>>.Empty;
        var builder = list.ToBuilder();
        for (var i = 0; i < 1000; i++)
        {
            builder.Add(Data<int>.Create(i));
        }

        SetOutputTo(builder.ToImmutable());
    }
}

public class ConsoleWorkUnit : WorkBase<int>
{
    public override void Work(ImmutableList<Data<int>> input)
    {
        foreach (var data in input)
        {
            Console.WriteLine("ConsoleWorkUnit printing {0}", data.Value);
        }
    }
}

public class SumPrinter : WorkBase<int>
{

    public override void Work(ImmutableList<Data<int>> input)
    {
        input.ToObservable().Buffer(2).Subscribe(PrintSum);
    }

    private void PrintSum(IList<Data<int>> obj)
    {
      Console.WriteLine("Sum of {0}, {1} is {2} ", obj.First().Value,obj.Last().Value ,obj.Sum(x=>x.Value) );
    }
}

These are run in a main like this

        var intgen = new IntGenerator();
        var cons = new ConsoleWorkUnit();
        var sumPrinter = new SumPrinter();

        intgen.ConnectOutputTo(cons);
        intgen.BufferOutputTo(3,sumPrinter);

        Task.Factory.StartNew(intgen.Work);

        Console.ReadLine();

Is this architecture sound?

Patterns for decorating private methods of a class

In the below class I have a public method called ProcessMessage. This method is responsible for processing the incoming messages. Processing a message involves different stage. I want to decorate this class in such a way that I can publish performance counter values from each stage of the message processing.

I know I can override the ProcessMessage method and rewrite the logic once again with publishing performance counter values. But is there any better way / pattern which I can apply, so that I don’t have to duplicate the logic once again in the decorated class.

public class MessageProcessor
{

    public void ProcessMessage()
    {
        ConvertReceivedMessage();
        SendToThirdParty();
        ReceiveResponse();
        ConvertResponseMessage();
        SendResponseToClient();
    }

    private void ConvertReceivedMessage()
    {
        //here I want to publish the performance counter value from the decorated class
    }
    private void SendToThirdParty()
    {
         //here I want to publish the performance counter value from the decorated class

    }
    private void ReceiveResponse()
    {
         //here I want to publish the performance counter value from the decorated class

    }
    private void ConvertResponseMessage()
    {
         //here I want to publish the performance counter value from the decorated class

    }

    private void SendResponseToClient()
    {
         //here I want to publish the performance counter value from the decorated class

    }

}

Thanks.

Is a workflow appropriate to import CSV, find and replace, normalize, then insert into db

I have a simple but tedious requirement, to build a system that each month imports several csv files into a database.

Each CSV file has different fields, and needs different sets of rules i.e.

  • column mapping
  • validation (i.e. do I have the correct number of fields, are they of the right type?)
  • capitalization
  • string replacement
  • regepr replacement

All of the above will be defined either at the file level or for individual fields.

I would like to use php (but it's not mandatory).

Since the requirement is pretty common, and it's quite a lot of work (I need to also code an editor for the rules), and I don't like reinventing the wheel, I am wondering if a workflow - based approach could prove satisfactory, leaving the user to choose which sets of rules are to be applied for each condition. Otherwise, please help me find the words that best describe this set of requirements, I am not mothertongue and I'm having a very hard time finding out best practices and alternative approaches, or at least some design-pattern level code to best implement this.

Right now I'm coding a javascript-editor for the rules that dynamically creates rules in json-format such as:

var rule = {
  ruleName:"Import from Agrabah",
  senderEmail:"jack@example.com",
  fileValidations: {
    format:"UTF-8",
    columns:8
  },
  ignoreLines: 1,
  find: ["Sir","Doc"],
  replace: ["Sr","Dr"],
  fields: [{
    index:0,
    column:"Name",
    case:"ucword"
  },{
    index:1,
    column:"Last",
    case:"ucword"
  },{
    index:2,
    column:"Province",
    case:"uc",
    find: ['ps'],
    replace: ['pu']
  }]
}

These are then saved in the php backend and used to define how each received file should be processed.

Each csv line is converted to an object through these rules, using the column names as properties of the object. Once this is done, a simple script updates/inserts the database record.

If my approach requires improvements or doesn't follow best practices please mention it

Please do not close this question, I am not asking for an opinion on 'the best product', rather the approach, patterns that may help me achieve a great result.

looping over different objects (that extends common super class) in one collection of superclasses

I struggle with inheritance in java. I have many elements that have similar properties like f.e. id,name or date. This elements have also specified parameters that belongs only to them, for example: email, address.

I would like to keeps this different elements in collection. And then display them in console, using loop. But when I put element like email (which is specified for concrete class User) it gives me an error:

value email is not a member of Person

Here is code:

public class Person {
  int id;
  String name;
  Date date;
}
public class User extends Person {
  String email;
  String login;
  String password;
}
public class Contact extends Person {
  Address address;
}

public class Customer {
  List<Person> persons;

  // AND NOW I WOULD LIKE TO PRINT THIS
  public void print() {
    for(Person person: this.persons) {
      System.out.println(person.id + " " + person.email);
    }
  }
}

I have many many similar problems with inheritance in Java. I don't know how to carry on with it? How to implement classes, that I could make it work.

Can You give me some good Design Pattern or clues?

Using design patterns in laravel 5

What are the design patterns that we can use in large projects with Laravel 5 ? is there any good tutorials or books to learn those design patterns ?

Non-OS Specific FD(File Descriptor) for C/C++

Linux is also treated as a file, a network socket. but, Windows is not. and common files and network sockets treated as "FD". if the code should not rely on the operating system, how should write?

i think it like below..

#ifndef INVALID_SOCKET
#define INVALID_SOCKET (-1)
#endif

class Descriptor {
private:
   int m_fd;

public:
   Descriptor() : m_fd(INVALID_SOCKET) { }
   virtual ~Descriptor() { this->close(); }
   virtual bool isValid();
   virtual bool close() = 0;
   virtual int getNo() { return m_fd; }
};

enum EListenFlags {
   E_LISTEN_READ = 1,
   E_LISTEN_WRITE = 2,
   E_LISTEN_ERROR = 4,
   E_LISTEN_NONBLOCK = 8
};

class AsyncDescriptor : public Descriptor {
// like EPoll (for linux) or IOCP (for Windows) or SELECT, POLL...
public:
   virtual bool listen(Descriptor* pDesc, int listenFlags) = 0;
   virtual bool dizzy(Descriptor* pDesc, int dizzyFlags) = 0;
   virtual bool wait(std::list<Descriptor*>& listOut) = 0;
   virtual bool list(std::list<Descriptor*>& listOut) = 0;
   virtual bool getFlags(Descriptor* pDesc, int* flagOutput) = 0;
};

class SocketDescriptor : public Descriptor {
     // Omitted.......
};

// Details are omitted below ...

How can i implement it???! :(

Running code in a PHP once

I'm trying to write a class in PHP that acts as a wrapper for a collection of command line tools to make them easier to use from PHP.

I have a single class (MyClass) in a file myclass.php.

I have code that checks to see if the required tools are installed and then sets a constant (TOOLS_AVAILABLE) to either true or false. Although it's not a lot of code, I only want it to run the first time somebody tries to instantiate my class. What's the best practice for handling this?