How to avoid Parallel Inheritance hierarchy in an API/Library

I'm working on an encryption library to simplify how my team uses encryption for api credentials and receiving encrypted messages from other teams/projects.

I started with these top-level interfaces to define the domain:

public interface Key {
    public byte[] getBytes();
}

public interface Message {
    public byte[] getBytes();
}

public interface Secret {
    public byte[] getBytes();
}

public interface Decrypter {
    public String decrypt(Secret secret, Key key);
}

public interface Encrypter {
    public Secret encrypt(Message message, Key key);
}

This worked well to wrap RSA encryption:

public class KeyPair {
    private final Key publicKey;
    private final Key privateKey;

    public Key getPublicKey() {
        return publicKey;
    }

    public Key getPrivateKey() {
        return privateKey;
    }
}

public class RsaEncrypter implements Encrypter {

    @Override
    public Secret encrypt(Message message, Key publicKey) {
        // Perform Encryption
    }
}

public class RsaDecrypter implements Decrypter {
    @Override
    public String decrypt(Secret secret, Key privateKey) {
        // Perform the decryption
    }
}

But now that I'm applying it to our AES Encryption use cases I've encountered a problem. The Secret contains an InitializationVector because we're using AES in CBC mode.

So I've got this:

public class AesSecret implements Secret {
    private byte[] cipherText;
    private byte[] initilizationVector;

    @Override
    public byte[] getBytes() {
        return cipherText;
    }

    public byte[] getInitilizationVector() {
        return initilizationVector;
    }
}

public class AesDecrypter implements Decrypter {
    @Override
    public String decrypt(Secret secret, Key key) {
        try {
            return decrypt((AesSecret) secret, key);
        } catch (ClassCastException e) {
            throw new IllegalArgumentException("AesDecrypter only supports subclasses of AesSecret", e);
        }
    }

    public String decrypt(AesSecret secret, Key key) {
        // Do the AES Decryption
    }
}

The ClassCastException makes me think that I'm violating the Liskov Substitution Principle, and introducing the parallel hierarchy code-smell. I've read the Visitor pattern is a common solution to this code smell, but I haven't figured out how it would apply to my situation.

Any suggestions? Or am I over-thinking this?