How to slim down a Fat Interface without breaking the Decorator pattern?

In my C++ library code I'm using an abstract base class as an interface to all different kinds of I/O-capable objects. It currently looks like this:

// All-purpose interface for any kind of object that can do I/O
class IDataIO
{
public:
   // basic I/O calls
   virtual ssize_t Read(void * buffer, size_t size) = 0;
   virtual ssize_t Write(const void * buffer, size_t size) = 0;

   // Seeking calls (implemented to return error codes
   // for I/O objects that can't actually seek)
   virtual result_t Seek(ssize_t offset, int whence) = 0;
   virtual ssize_t GetCurrentSeekPosition() const = 0;
   virtual ssize_t GetStreamLength() const = 0;

   // Packet-specific calls (implemented to do nothing
   // for I/O objects that aren't packet-oriented)
   virtual const IPAddressAndPort & GetSourceOfLastReadPacket() const = 0;
   virtual result_t SetPacketSendDestination(const IPAddressAndPort & iap) = 0;
};

This works pretty well -- I have various concrete subclasses for TCP, UDP, memory buffers, SSL, RS232, and so on, and I'm able to write I/O-agnostic routines that can be used in conjunction with any of them.

I also have various decorator classes that take ownership of an existing IDataIO object and serve as a behavior-modifying front-end to that object. These decorator classes are useful because a single decorator class can modify the behavior of any IDataIO object. Here's a simple (toy) example:

/** Example decorator class:  This object wraps any given
  * child IDataIO object, such that all data going out is
  * obfuscated by applying an XOR transformation to the bytes,
  * and any data coming in is de-obfuscated the same way.
  */
class XorDataIO : public IDataIO
{
public:
   XorDataIO(IDataIO * child) : _child(child) {/* empty */}
   virtual ~XorDataIO() {delete _child;}

   virtual ssize_t Read(void * buffer, size_t size)
   {
      ssize_t ret = _child->Read(buffer, size);
      if (ret > 0) XorData(buffer, ret);
      return ret;
   }

   virtual ssize_t Write(const void * buffer, size_t size)
   {
      XorData(buffer, size);   // pseudocode here, but you get the idea
      return _child->Write(buffer, size);
   }

   virtual result_t Seek(ssize_t offset, int whence) {return _child->Seek(offset, whence);}
   virtual ssize_t GetCurrentSeekPosition() const    {return _child->GetCurrentSeekPosition();}
   virtual ssize_t GetStreamLength() const           {return _child->GetStreamLength();}

   virtual const IPAddressAndPort & GetSourceOfLastReadPacket() const      {return _child->GetSourceOfLastReadPacket();}
   virtual result_t SetPacketSendDestination(const IPAddressAndPort & iap) {return _child->SetPacketSendDestination(iap);}

private:
   IDataIO * _child;
};

This is all well and good, but what's bothering me is that my IDataIO class looks like an example of a fat interface -- for example, a UDPSocketDataIO class will never be able to implement the Seek()/GetCurrentSeekPosition()/GetStreamLength() methods, while a FileDataIO class will never be able to implement the GetSourceOfLastReadPacket()/SetPacketSendDestination() methods. So both classes are forced to implement those methods as stubs that just do nothing and return an error code -- which works, but it's ugly.

To solve the problem, I'd like to break out the IDataIO interface into separate chunks, like this:

// The bare-minimum interface for any object that we can
// read bytes from, or write bytes to (e.g. TCP or RS232)
class IDataIO
{
public:
   virtual ssize_t Read(void * buffer, size_t size) = 0;
   virtual ssize_t Write(const void * buffer, size_t size) = 0;
};

// A slightly extended interface for objects (e.g. files
// or memory-buffers) that also allows us to seek to a
// specified offset within the data-stream.
class ISeekableDataIO : public IDataIO
{
public:
   virtual result_t Seek(ssize_t offset, int whence) = 0;
   virtual ssize_t GetCurrentSeekPosition() const = 0;
   virtual ssize_t GetStreamLength() const = 0;
};

// A slightly extended interface for packet-oriented
// objects (e.g. UDP sockets)
class IPacketDataIO : public IDataIO
{
public:
   virtual const IPAddressAndPort & GetSourceOfLastReadPacket() const = 0;
   virtual result_t SetPacketSendDestination(const IPAddressAndPort & iap) = 0;
};

.... so now I can subclass UDPSocketDataIO from the IPacketDataIO sub-interface, and subclass FileDataIO from the ISeekableDataIO interface, while TCPSocketDataIO can still subclass directly from DataIO, and so on. That way each type of I/O object presents the interface only to the functionality it can actually support, and nobody has to implement no-op/stub versions of methods that are irrelevant to what they do.

The problem that arises, then is with the decorators -- what interface should my XorDataIO subclass inherit from in this scenario? I suppose I could write a XorDataIO, a XorSeekableDataIO, and a XorPacketDataIO, so that all three types of interface can be fully decorated, but I'd really rather not -- that seems like a lot of overhead, particularly if I have multiple different adapter classes already and I don't want to multiple their number by three.

Is there some well-known clever/elegant way to address this problem, such that I can have my cake and eat it too?