Click here to Skip to main content
15,444,626 members
Please Sign up or sign in to vote.
1.00/5 (1 vote)
See more: , +
I have created a trading environment using tfagent
env = TradingEnv(df=df.head(100000), lkb=1000)
tf_env = tf_py_environment.TFPyEnvironment(env)

and passed a df of 100000 rows from which only closing prices are used which a numpy array of 100000 stock price time series data
df: Date Open High Low Close volume
0 2015-02-02 09:15:00+05:30 586.60 589.70 584.85 584.95 171419
1 2015-02-02 09:20:00+05:30 584.95 585.30 581.25 582.30 59338
2 2015-02-02 09:25:00+05:30 582.30 585.05 581.70 581.70 52299
3 2015-02-02 09:30:00+05:30 581.70 583.25 581.70 582.60 44143
4 2015-02-02 09:35:00+05:30 582.75 584.00 582.75 582.90 42731
... ... ... ... ... ... ...
99995 2020-07-06 11:40:00+05:30 106.85 106.90 106.55 106.70 735032
99996 2020-07-06 11:45:00+05:30 106.80 107.30 106.70 107.25 1751810
99997 2020-07-06 11:50:00+05:30 107.30 107.50 107.10 107.35 1608952
99998 2020-07-06 11:55:00+05:30 107.35 107.45 107.10 107.20 959097
99999 2020-07-06 12:00:00+05:30 107.20 107.35 107.10 107.20 865438

at each step the agent has access to previous 1000 prices + current price of stock = 1001 and it can take 3 possible action from 0,1,2

then I wrapped it in TFPyEnvironment to convert it to tf_environment

the prices that the agent can observe is a 1d numpy array
prices = [584.95 582.3 581.7 ... 107.35 107.2 107.2 ]


TimeStep Specs
TimeStep Specs: TimeStep( {'discount': BoundedTensorSpec(shape=(), dtype=tf.float32, name='discount', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32)), 'observation': BoundedTensorSpec(shape=(1001,), dtype=tf.float32, name='_observation', minimum=array(0., dtype=float32), maximum=array(3.4028235e+38, dtype=float32)), 'reward': TensorSpec(shape=(), dtype=tf.float32, name='reward'), 'step_type': TensorSpec(shape=(), dtype=tf.int32, name='step_type')}) Action Specs: BoundedTensorSpec(shape=(), dtype=tf.int32, name='_action', minimum=array(0, dtype=int32), maximum=array(2, dtype=int32))

then I build a dqn agent but I want to build it with a Conv1d layer

my network consist of
Conv1D,
MaxPool1D,
Conv1D,
MaxPool1D,
Dense_64,
Dense_32 ,
q_value_layer

I created a list layers using tf.keras.layers api and stored it in dense_layers list and created a Sequential Network

DQN_Agent
`learning_rate = 1e-3

action_tensor_spec = tensor_spec.from_spec(tf_env.action_spec())
num_actions = action_tensor_spec.maximum - action_tensor_spec.minimum + 1

dense_layers = []

dense_layers.append(tf.keras.layers.Conv1D(
64,
kernel_size=(10),
activation=tf.keras.activations.relu,
input_shape=(1,1001),
))

dense_layers.append(
tf.keras.layers.MaxPool1D(
pool_size=2,
strides=None,
padding='valid',
))

dense_layers.append(tf.keras.layers.Conv1D(
64,
kernel_size=(10),
activation=tf.keras.activations.relu,
))

dense_layers.append(
tf.keras.layers.MaxPool1D(
pool_size=2,
strides=None,
padding='valid',
))

dense_layers.append(
tf.keras.layers.Dense(
64,
activation=tf.keras.activations.relu,
))

dense_layers.append(
tf.keras.layers.Dense(
32,
activation=tf.keras.activations.relu,
))

q_values_layer = tf.keras.layers.Dense(
num_actions,
activation=None,
kernel_initializer=tf.keras.initializers.RandomUniform(
minval=-0.03, maxval=0.03),
bias_initializer=tf.keras.initializers.Constant(-0.2))

q_net = sequential.Sequential(dense_layers + [q_values_layer])`

`optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)

train_step_counter = tf.Variable(0)

agent = dqn_agent.DqnAgent(
tf_env.time_step_spec(),
tf_env.action_spec(),
q_network=q_net,
optimizer=optimizer,
td_errors_loss_fn=common.element_wise_squared_loss,
train_step_counter=train_step_counter)

agent.initialize()`

but when I passed the q_net as a q_network to DqnAgent I came across this error
`---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in ()
68 optimizer=optimizer,
69 td_errors_loss_fn=common.element_wise_squared_loss,
---> 70 train_step_counter=train_step_counter)
71
72 agent.initialize()

7 frames
/usr/local/lib/python3.7/dist-packages/tf_agents/networks/sequential.py in call(self, inputs, network_state, **kwargs)
222 else:
223 # Does not maintain state.
--> 224 inputs = layer(inputs, **layer_kwargs)
225
226 return inputs, tuple(next_network_state)

ValueError: Exception encountered when calling layer "sequential_54" (type Sequential).

Input 0 of layer "conv1d_104" is incompatible with the layer: expected min_ndim=3, found ndim=2. Full shape received: (1, 1001)

Call arguments received by layer "sequential_54" (type Sequential):
• inputs=tf.Tensor(shape=(1, 1001), dtype=float32)
• network_state=()
• kwargs={'step_type': 'tf.Tensor(shape=(1,), dtype=int32)', 'training': 'None'}
In call to configurable 'DqnAgent' (<class 'tf_agents.agents.dqn.dqn_agent.dqnagent'="">)`


What I have tried:

I know it has something to do with the input shape of first layer of cov1d but cant figure out what am doing wrong

at each time_step the agent is receiving a observation of prices of 1d array of length 1001 then the input shape of conv1d should be (1,1001) but its wrong and I don't know how to solve this error

need help
Posted
Updated 20-Jun-22 0:12am
v2
Comments
ashish bhong 20-Jun-22 7:31am     CRLF
do you know how to solve this problem
Richard Deeming 20-Jun-22 7:32am     CRLF
Yes: there's a great big "Delete" button on your question. Press it. And in future, don't post the same question in multiple places on the site site.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900