dimanche 18 août 2019

parrellel processing with abstract class

I have an abstract class which contains an abstract method which is needed to process data. Each class which inherits from the abstract class should have the same procedure to prepare the data and utilize multiprocessing to start up the processing function; however, the abstract method which does the processing on each file is unique to the child class.

This causes a problem of pickling, as seen below.

import numpy as np
from abc import ABC, abstractmethod
from p_tqdm import p_map


class Parent(ABC):
    def __init__(self, data):
        self.data = data

    def _process_data(self):
        for d in p_map(self._func, self.data):  # <--- issue in pickling 
            print(d)

    @abstractmethod
    def _func(self):
        # Each `Child` class (Any class that inherits `Parent`)
        # has a unique function to process the data
        pass

class Child(Parent):
    def __init__(self, data):
        super().__init__(data)

    def _func(self, data):
        return data + 1 # do something to data

class AnotherChild(Parent):
    def __init__(self, data):
        super().__init__(data)

    def _func(self, data):
        return data * 10 # do something else to data


c = Child(data=np.array([0,1,2,3,4,5]))
c._process_data()

a = AnotherChild(data=np.array([0,1,2,3,4,5]))
a._process_data()

receives error:

TypeError: can't pickle _abc_data objects

Any suggestions as to how to achieve the goal of inheriting the multiprocessing and preparation of data but calling specific functions for each child class?

Aucun commentaire:

Enregistrer un commentaire