Postgres: Why do I get an input syntax error when modifying a char column into a timestamp?

I copied data from a csv to my table. However, the timestamp data was copied with the column on the table as char(40). Now I want to modify the table to have it as a timestamp but keep getting an error specifying input syntax error.

The data is formatted in ISO8601 I believe

e.g. 2016-06-03T08:00:00.020584124-04:00

I've tried both of these SQL statements with the same result

ALTER TABLE public.data
    ALTER COLUMN timestamp TYPE TIMESTAMP WITH TIME ZONE USING timestamp::TIMESTAMP WITH TIME ZONE;

ALTER TABLE public.data
   ALTER COLUMN timestamp TYPE TIMESTAMPTZ USING timestamp::TIMESTAMPTZ;