I have an abstract class which contains an abstract method which is needed to process data. Each class which inherits from the abstract class should have the same procedure to prepare the data and utilize multiprocessing to start up the processing function; however, the abstract method which does the processing on each file is unique to the child class.
This causes a problem of pickling, as seen below.
import numpy as np
from abc import ABC, abstractmethod
from p_tqdm import p_map
class Parent(ABC):
def __init__(self, data):
self.data = data
def _process_data(self):
for d in p_map(self._func, self.data): # <--- issue in pickling
print(d)
@abstractmethod
def _func(self):
# Each `Child` class (Any class that inherits `Parent`)
# has a unique function to process the data
pass
class Child(Parent):
def __init__(self, data):
super().__init__(data)
def _func(self, data):
return data + 1 # do something to data
class AnotherChild(Parent):
def __init__(self, data):
super().__init__(data)
def _func(self, data):
return data * 10 # do something else to data
c = Child(data=np.array([0,1,2,3,4,5]))
c._process_data()
a = AnotherChild(data=np.array([0,1,2,3,4,5]))
a._process_data()
receives error:
TypeError: can't pickle _abc_data objects
Any suggestions as to how to achieve the goal of inheriting the multiprocessing and preparation of data but calling specific functions for each child class?
Aucun commentaire:
Enregistrer un commentaire