Yugon
/
RL_exhibit
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

State switching. Needs refactoring

Browse Source
development
Ugo Finnendahl 5 years ago
parent 088bb10805
commit 12a5cb383e
7 changed files with 751 additions and 305 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 32
      css/src/style.scss
  2. 77
      css/style.css
  3. 2
      css/style.min.css
  4. 58
      index.html
  5. 2
      js/controls.js
  6. 30
      js/rl.js
  7. 851
      js/view.js

32
css/src/style.scss
Unescape View File

@ -3,6 +3,8 @@
padding: 0;
}
body{
margin: 0;
padding: 0;
// background-color: ;
font-family: sans-serif;
}
@ -13,11 +15,13 @@ body{
#canvas{
height: 100%;
}
nav{
position: absolute;
top: 10px;
left: 10px;
}
.absolute{
position: absolute;
top:0;
@ -34,13 +38,37 @@ nav{
.sliders{
position: absolute;
top: 7vh;
top: 20vh;
left: 2vw;
width: 20vw;
}
#formula{
position: absolute;
top: -5vh;
top: 71.5vh;
width: 90vw;
}
.score{
position: absolute;
top: 7vh;
left: 2vw;
width: 20vw;
}
.lightbox{
padding: 2em;
position: absolute;
top:0;
left: 50%;
transform: translateX(-50%) translateY(-100%);
z-index: 10;
width: 50%;
height: 40%;
background-color: #BBB;
transition: all 1s;
&.active{
top:50%;
transform: translateX(-50%) translateY(-50%);
}
}

77
css/style.css
Unescape View File

@ -0,0 +1,77 @@
* {
margin: 0;
padding: 0;
}
body {
margin: 0;
padding: 0;
font-family: sans-serif;
}
#container {
height: 100vh;
position: relative;
}
#canvas {
height: 100%;
}
nav {
position: absolute;
top: 10px;
left: 10px;
}
.absolute {
position: absolute;
top: 0;
left: 0;
}
.plot {
position: absolute;
top: 2vh;
right: 2vw;
width: 20vw;
height: 10vw;
}
.sliders {
position: absolute;
top: 20vh;
left: 2vw;
width: 20vw;
}
#formula {
position: absolute;
top: 71.5vh;
width: 90vw;
}
.score {
position: absolute;
top: 7vh;
left: 2vw;
width: 20vw;
}
.lightbox {
padding: 2em;
position: absolute;
top: 0;
left: 50%;
transform: translateX(-50%) translateY(-100%);
z-index: 10;
width: 50%;
height: 40%;
background-color: #BBB;
transition: all 1s;
}
.lightbox.active {
top: 50%;
transform: translateX(-50%) translateY(-50%);
}

2
css/style.min.css vendored
Unescape View File

@ -1 +1 @@
*{margin:0;padding:0}body{font-family:sans-serif}#container{height:100vh;position:relative}#canvas{height:100%}nav{position:absolute;top:10px;left:10px}.absolute{position:absolute;top:0;left:0}.plot{position:absolute;top:2vh;right:2vw;width:20vw;height:10vw}.sliders{position:absolute;top:7vh;left:2vw;width:20vw}#formula{position:absolute;top:-5vh;width:90vw}
*{margin:0;padding:0}body{margin:0;padding:0;font-family:sans-serif}#container{height:100vh;position:relative}#canvas{height:100%}nav{position:absolute;top:10px;left:10px}.absolute{position:absolute;top:0;left:0}.plot{position:absolute;top:2vh;right:2vw;width:20vw;height:10vw}.sliders{position:absolute;top:20vh;left:2vw;width:20vw}#formula{position:absolute;top:71.5vh;width:90vw}.score{position:absolute;top:7vh;left:2vw;width:20vw}.lightbox{padding:2em;position:absolute;top:0;left:50%;transform:translateX(-50%) translateY(-100%);z-index:10;width:50%;height:40%;background-color:#BBB;transition:all 1s}.lightbox.active{top:50%;transform:translateX(-50%) translateY(-50%)}

58
index.html
Unescape View File

@ -16,47 +16,45 @@
<!-- The loading of KaTeX is deferred to speed up page rendering -->
<script src="https://cdn.jsdelivr.net/npm/katex@0.11.1/dist/katex.min.js" integrity="sha384-y23I5Q6l+B6vatafAwxRu/0oK/79VlbSz7Q9aiSZUvyWYIYsd+qj+o24G5ZU2zJz" crossorigin="anonymous"></script>
<title>RL exhibit - prototype</title>
<link rel="stylesheet" href="https://gitcdn.xyz/cdn/jzilg/embellish.css/fff9961c5fec2d1c8ff53c87b12e18d5c8db7761/embellish.min.css">
<link rel="stylesheet" href="css/style.min.css">
</head>
<body>
<div id="app">
<v-stage ref="stage" :config="stage_config">
<v-layer ref="map_layer">
<v-group ref="mini_map_group" :config="mini_map_config">
<v-group v-for="(t_type, idx) in maze.map.flat()" :config="get_field_config(idx)">
<v-rect :config="get_tile_config(idx, t_type)"></v-rect>
<v-image :config="energy_config" v-if="t_type==tile"></v-image>
</v-group>
<v-group v-for="(action,idx) in q_table" :config="get_field_config(idx)">
<v-shape v-for="(value, key) in action" :config="get_triangle_config(value, key)"></v-shape>
<v-text v-for="i in 4" :config="get_q_text_config(action,i)"></v-text>
</v-group>
<v-image :config="robot_config"></v-image>
</v-group>
</v-layer>
<light-box :content="lightText" :options="lightOptions" :active="lightpopup"></light-box>
<v-stage ref="stage" :config="stage_config" v-show="isActive('global')">
<rl-map :machine="machine" :maze="maze" :config="map_config"></rl-map>
</v-stage>
<v-stage ref="stage" :config="stage_config" v-show="isActive('local')">
<rl-local :machine="machine" :maze="maze" :config="map_config"></rl-local>
</v-stage>
<line-chart css-classes="plot" :chart-data="datacollection" :options="plot_options"></line-chart>
<line-chart css-classes="plot" :chart-data="datacollection" :options="plot_options" v-show="isActive('plot')"></line-chart>
<div class="sliders">
<h1>Learning Rate {{learning_rate}}</h1>
<vue-slider v-model="learning_rate" :drag-on-click="true" v-bind="slider_config"></vue-slider>
<h1>Discount Factor {{discount_factor}}</h1>
<vue-slider v-model="discount_factor" :drag-on-click="true" v-bind="slider_config"></vue-slider>
<h1>Epsilon {{epsilon}}</h1>
<vue-slider v-model="epsilon" :drag-on-click="true" v-bind="slider_config"></vue-slider>
<h1>Current Score</h1>
<h2>{{score}}</h2>
<div class="sliders" v-show="isActive('sliders')">
<h1>Learning Rate {{machine.learning_rate}}</h1>
<vue-slider v-model="machine.learning_rate" :drag-on-click="true" v-bind="slider_config"></vue-slider>
<h1>Discount Factor {{machine.discount_factor}}</h1>
<vue-slider v-model="machine.discount_factor" :drag-on-click="true" v-bind="slider_config"></vue-slider>
<h1>Epsilon {{machine.epsilon}}</h1>
<vue-slider v-model="machine.epsilon" :drag-on-click="true" v-bind="slider_config"></vue-slider>
<div id="formula"></div>
</div>
<div class="score" v-show="isActive('score')">
<h1>Current Score</h1>
<h2>{{machine.score}}</h2>
</div>
<nav>
<button class="button" onclick="machine.run(1)">run 1 episode!</button>
<button class="button" onclick="machine.run(100)">run 100 episodes!</button>
<button class="button" onclick="machine.auto_step();">auto step!</button>
<button class="button" onclick="machine.greedy_step();">greedy step!</button>
<button class="button" onclick="machine.reset_machine()">reset machine</button>
<nav class="navi" v-show="isActive('navi')">
<button v-for="(item, key) in navigation" v-on:click="item">{{ key }}</button>
</nav>
</div>
<script>
var map = [
[0, 0, 1, 8, 0, 0, 0, 0, 0, 0],

2
js/controls.js vendored
Unescape View File

@ -27,7 +27,7 @@ function key_callback(e) {
break;
}
var ret = 1;
if (tmp != undefined){
if (tmp != undefined && document.querySelector(".lightbox.active") == null){
ret = machine.step(tmp);
}
// show_q_table();

30
js/rl.js
Unescape View File

@ -4,6 +4,7 @@ class RL_machine {
rewards,
start_state,
end_states,
start_score,
end_score,
learning_rate,
discount_factor,
@ -14,6 +15,7 @@ class RL_machine {
this.lr = learning_rate;
this.df = discount_factor;
this.start_state = start_state;
this.start_score = start_score;
this.end_score = end_score;
this.end_states = end_states;
this.epsilon = epsilon;
@ -27,17 +29,17 @@ class RL_machine {
}
}
this.episode = 0;
this.state = this.start_state;
this.score = 0;
this.running = false;
this.score_history = [];
this.state = this.start_state;
this.score = this.start_score;
}
new_episode(){
// add_new_episode_callback
this.episode++;
this.state = this.start_state;
this.score_history.push(this.score);
this.score = 0;
this.state = this.start_state;
this.score = this.start_score;
}
auto_step(){
if (Math.random() < this.epsilon){
@ -52,7 +54,23 @@ class RL_machine {
step(action){
this.state = this.update_q_table(this.state, action);
// add_new_step_callback
if (this.end_states.indexOf(this.state) >= 0 || this.score < this.end_score){
if (this.end_states.indexOf(this.state) >= 0) {
var succ_event = new CustomEvent("episode",{
detail: "success"
});
if (!this.running) {
window.dispatchEvent(succ_event);
}
this.new_episode();
return 2
}
if (this.score <= this.end_score){
var fail_event = new CustomEvent("episode",{
detail: "failed"
});
if (!this.running) {
window.dispatchEvent(fail_event);
}
this.new_episode();
return 2
}
@ -191,4 +209,4 @@ var maze = new Maze(map, reward);
var learning_rate = 0.75;
var discount_factor = 0.8;
var machine = new RL_machine(maze.actions, maze.transactions, maze.rewards, maze.start_state, maze.end_states, -999, learning_rate, discount_factor, 0.2);
var machine = new RL_machine(maze.actions, maze.transactions, maze.rewards, maze.start_state, maze.end_states, 50, 0, learning_rate, discount_factor, 0.2);

851
js/view.js
Unescape View File

@ -1,31 +1,6 @@
Vue.component('line-chart', {
extends: VueChartJs.Line,
mixins: [VueChartJs.mixins.reactiveProp],
props: ['options'],
// mixins: [VueChartJs.mixins.reactiveData],
// props: ['options','labels', 'datasets'],
// watch: {
// 'labels': function(new_val) {
// this.chartData = {
// 'labels': new_val,
// 'datasets': this.datasets};
// },
// 'datasets': {
// deep:true,
// handler: function(new_val) {
// this.chartData = {
// 'labels': this.labels,
// 'datasets': new_val};
// }
// }
// },
mounted() {
this.renderChart(this.chartData, this.options);
},
})
var palette = ['#d2000d', '#d30512', '#d40a17', '#d50f1c', '#d61420', '#d71a25', '#d71f2a', '#d8242f', '#d92934', '#da2e39', '#db333d', '#dc3842', '#dd3d47', '#de424c', '#df4751', '#e04d56', '#e0525a', '#e1575f', '#e25c64', '#e36169', '#e4666e', '#e56b73', '#e67077', '#e7757c', '#e87a81', '#e98086', '#e9858b', '#ea8a90', '#eb8f95', '#ec9499', '#ed999e', '#ee9ea3', '#efa3a8', '#f0a8ad', '#f1adb2', '#f2b3b6', '#f2b8bb', '#f3bdc0', '#f4c2c5', '#f5c7ca', '#f6cccf', '#f7d1d3', '#f8d6d8', '#f9dbdd', '#fae0e2', '#fbe6e7', '#fbebec', '#fcf0f0', '#fdf5f5', '#fefafa', '#ffffff', '#fafcfa', '#f5f9f5', '#f0f6f0', '#ebf3ec', '#e6f1e7', '#e1eee2', '#dcebdd', '#d7e8d8', '#d3e5d3', '#cee2cf', '#c9dfca', '#c4dcc5', '#bfd9c0', '#bad6bb', '#b5d4b6', '#b0d1b2', '#abcead', '#a6cba8', '#a1c8a3', '#9cc59e', '#97c299', '#92bf95', '#8dbc90', '#88b98b', '#84b786', '#7fb481', '#7ab17c', '#75ae77', '#70ab73', '#6ba86e', '#66a569', '#61a264', '#5c9f5f', '#579c5a', '#529a56', '#4d9751', '#48944c', '#439147', '#3e8e42', '#398b3d', '#348839', '#308534', '#2b822f', '#267f2a', '#217d25', '#1c7a20', '#17771c', '#127417', '#0d7112', '#086e0d']
// ----------------------------------------------------------------------------
// -------------------------------- Plot --------------------------------------
// ----------------------------------------------------------------------------
Array.prototype.simpleSMA = function(N) {
return this.map(
@ -55,201 +30,66 @@ Array.prototype.max = function() {
});
};
app = new Vue({
el: '#app',
components: {
VueSlider: window['vue-slider-component']
},
data: {
width: 0,
height: 0,
q_table: machine.q_table,
maze: maze,
state: {
x: 0,
y: 0
},
state_tween: new TimelineLite(),
score: machine.score,
score_history: machine.score_history,
labels: [],
learning_rate: machine.lr,
discount_factor: machine.df,
epsilon: machine.epsilon,
slider_config: {
min: 0,
max: 1,
duration: 0,
interval: 0.01,
tooltip: 'none'
},
robot_image: null,
energy_image: null,
},
created() {
// Resize handler
window.addEventListener('resize', this.handleResize)
this.handleResize();
// State wrapper
var s = machine.state;
var $this = this;
this.state = this.s2p(s);
Object.defineProperty(machine, 'state', {
get: function() {
return this._state
},
set: function(ne) {
this._state = ne;
$this.handleState(this._state);
}
});
machine.state = s;
// Score wrapper
var s = machine.score;
var $this = this;
this.score = s;
Object.defineProperty(machine, 'score', {
get: function() {
return this._score
},
set: function(ne) {
this._score = ne;
$this.score = ne
}
});
machine.score = s;
// Score history wrapper
var s = machine.score_history;
var $this = this;
this.score_history = s;
Object.defineProperty(machine, 'score_history', {
get: function() {
return this._score_history
Vue.component('line-chart', {
extends: VueChartJs.Line,
mixins: [VueChartJs.mixins.reactiveProp],
props: ['options'],
mounted() {
this.renderChart(this.chartData, this.options);
},
set: function(ne) {
this._score_history = ne;
$this.score_history = ne
}
});
machine.score_history = s;
})
// ----------------------------------------------------------------------------
// --------------------------------- Map --------------------------------------
// ----------------------------------------------------------------------------
function set_images($this){
const robot_image = new window.Image();
robot_image.src = "img/robot.png";
// robot_image.src = "https://konvajs.org/assets/yoda.jpg";
robot_image.onload = () => {
// set image only when it is loaded
this.robot_image = robot_image;
$this.robot_image = robot_image;
};
const energy_image = new window.Image();
energy_image.src = "img/station.png";
// energy_image.src = "https://konvajs.org/assets/yoda.jpg";
energy_image.onload = () => {
// set image only when it is loaded
this.energy_image = energy_image;
$this.energy_image = energy_image;
};
},
destroyed() {
window.removeEventListener('resize', this.handleResize)
},
computed: {
datacollection: function() {
return {
labels: Array.from(Array(this.score_history.length).keys()),
datasets: [{
label: 'Data One',
backgroundColor: 'rgb(0,0,0,0)',
data: this.score_history,//.simpleSMA(Math.round(50)),
fill: false,
borderColor: 'rgb(255, 159, 64)',
pointRadius: 1,
},
// {
// label: 'Data One',
// backgroundColor: 'rgb(0,0,0,0)',
// data: this.score_history.max(),
// fill: false,
// borderColor: 'rgb(64, 159, 255)',
// pointRadius: 1,
// },
]
}
},
plot_options: function() {
var $this = this;
return {
responsive: true,
maintainAspectRatio: false,
scales: {
xAxes: [{
// type: 'linear',
ticks: {
maxTicksLimit: 8,
maxRotation: 0,
}
}]
},
legend: {
display: false
}
}
},
stage_config: function() {
return {
width: this.width,
height: this.height,
}
},
mini_map_config: function() {
}
var palette = ['#d2000d', '#d30512', '#d40a17', '#d50f1c', '#d61420', '#d71a25', '#d71f2a', '#d8242f', '#d92934', '#da2e39', '#db333d', '#dc3842', '#dd3d47', '#de424c', '#df4751', '#e04d56', '#e0525a', '#e1575f', '#e25c64', '#e36169', '#e4666e', '#e56b73', '#e67077', '#e7757c', '#e87a81', '#e98086', '#e9858b', '#ea8a90', '#eb8f95', '#ec9499', '#ed999e', '#ee9ea3', '#efa3a8', '#f0a8ad', '#f1adb2', '#f2b3b6', '#f2b8bb', '#f3bdc0', '#f4c2c5', '#f5c7ca', '#f6cccf', '#f7d1d3', '#f8d6d8', '#f9dbdd', '#fae0e2', '#fbe6e7', '#fbebec', '#fcf0f0', '#fdf5f5', '#fefafa', '#ffffff', '#fafcfa', '#f5f9f5', '#f0f6f0', '#ebf3ec', '#e6f1e7', '#e1eee2', '#dcebdd', '#d7e8d8', '#d3e5d3', '#cee2cf', '#c9dfca', '#c4dcc5', '#bfd9c0', '#bad6bb', '#b5d4b6', '#b0d1b2', '#abcead', '#a6cba8', '#a1c8a3', '#9cc59e', '#97c299', '#92bf95', '#8dbc90', '#88b98b', '#84b786', '#7fb481', '#7ab17c', '#75ae77', '#70ab73', '#6ba86e', '#66a569', '#61a264', '#5c9f5f', '#579c5a', '#529a56', '#4d9751', '#48944c', '#439147', '#3e8e42', '#398b3d', '#348839', '#308534', '#2b822f', '#267f2a', '#217d25', '#1c7a20', '#17771c', '#127417', '#0d7112', '#086e0d']
Vue.component('rl-map', {
props: ['machine', 'maze', 'config'],
data: function () {
return {
x: this.width / 2 - (this.base_size * (this.maze.width) / 2),
y: this.height / 2 - (this.base_size * (this.maze.height) / 2),
scale: {
x: 1,
y: 1
}
robot_image: null,
energy_image: null,
}
},
local_layer: function() {
return {
x: this.width / 2,
y: this.height / 2,
scale: {
x: 2,
y: 2
}
}
created() {
set_images(this);
},
map_config: function() {
computed: {
main_config: function(){
return {
x: this.base_size * (this.maze.width - this.state.x),
y: this.base_size * (this.maze.height - this.state.y),
...this.config,
offset: {
x: this.base_size * this.maze.width + this.base_size / 2,
y: this.base_size * this.maze.height + this.base_size / 2,
}
x: -(this.config.width-this.base_size*this.maze.width)/2,
y: -(this.config.height-this.base_size*this.maze.height)/2,
}
},
agent_config: function() {
return {
sides: 5,
radius: this.base_size / 3,
fill: '#00D2FF',
stroke: 'black',
strokeWidth: this.strokeW,
offset: {
x: -this.base_size / 2,
y: -this.base_size / 2
},
x: this.base_size * this.state.x,
y: this.base_size * this.state.y,
}
},
robot_config: function() {
return {
height: this.base_size,
width: this.base_size,
x: this.base_size * this.state.x,
y: this.base_size * this.state.y,
x: this.base_size * this.machine.state.x,
y: this.base_size * this.machine.state.y,
image: this.robot_image,
}
},
@ -265,7 +105,7 @@ app = new Vue({
}
},
base_size: function() {
return Math.min(this.stage_config.height * 0.8 / this.maze.height, this.stage_config.width * 0.5 / this.maze.width);
return Math.min(this.config.height/this.maze.height, this.config.width/this.maze.width);
},
strokeW: function() {
return this.base_size / 50;
@ -286,45 +126,8 @@ app = new Vue({
}
},
methods: {
s2p: function(state) {
return {
x: (state % this.maze.width),
y: Math.floor(state / this.maze.width),
}
},
p2s: function(x, y) {
return x + y * this.maze.width;
},
handleResize: function() {
this.width = window.innerWidth;
this.height = window.innerHeight;
},
handleState: function(s) {
if (!machine.running) {
this.state_tween.to(this.state, 0.2, {
x: this.s2p(s).x,
y: this.s2p(s).y
});
} else {
this.state = this.s2p(s);
}
// this.hidden_state = s;
},
get_grid_line_config: function(idx, y = false) {
var offset = this.strokeW / 2;
if (y) {
var points = [-offset, Math.round(idx * this.base_size), this.base_size * this.maze.width + offset, Math.round(idx * this.base_size)];
} else {
var points = [Math.round(idx * this.base_size), -offset, Math.round(idx * this.base_size), this.base_size * this.maze.height + offset];
}
return {
points: points,
stroke: '#ddd',
strokeWidth: this.strokeW,
}
},
get_tile_type: function(state) {
var pos = this.s2p(state);
var pos = this.machine.s2p(state);
if (pos.y > maze.height) {
return null;
} else if (pos.x > maze.width) {
@ -333,14 +136,8 @@ app = new Vue({
return maze.map[pos.y][pos.x];
}
},
in_plus: function(pos1, pos2) {
if (Math.abs(pos1.x - pos2.x) + Math.abs(pos1.y - pos2.y) < 2) {
return true;
}
return false;
},
get_field_config: function(state) {
var pos = this.s2p(state);
var pos = this.machine.s2p(state);
return {
x: this.base_size * pos.x+this.base_size/2,
y: this.base_size * pos.y+this.base_size/2,
@ -443,20 +240,20 @@ app = new Vue({
}
},
get_tile_config: function(i, t_type, local = false) {
var pos = this.s2p(i);
// var pos = this.s2p(i);
var over = {};
// not in plus
if (local) {
if (!this.in_plus(this.s2p(i), {
x: Math.round(this.state.x),
y: Math.round(this.state.y)
if (!this.in_plus(this.machine.s2p(i), {
x: Math.round(this.machine.state.x),
y: Math.round(this.machine.state.y)
})) {
over = {
opacity: 0,
fill: "#eee"
};
} else if (i != this.p2s(Math.round(this.state.x), Math.round(this.state.y))) {
} else if (i != this.p2s(Math.round(this.machine.state.x), Math.round(this.machine.state.y))) {
over = {
opacity: 1,
fill: "#eee"
@ -510,25 +307,553 @@ app = new Vue({
...over,
}
}
},
},
template:
`<v-layer ref="map_layer">
<v-group ref="map_group" :config="main_config">
<v-group v-for="(t_type, idx) in maze.map.flat()" :config="get_field_config(idx)">
<v-rect :config="get_tile_config(idx, t_type)"></v-rect>
<v-image :config="energy_config" v-if="t_type==8"></v-image>
</v-group>
<v-group v-for="(action,idx) in machine.q_table" :config="get_field_config(idx)">
<v-shape v-for="(value, key) in action" :config="get_triangle_config(value, key)"></v-shape>
<v-text v-for="i in 4" :config="get_q_text_config(action,i)"></v-text>
</v-group>
<v-image :config="robot_config"></v-image>
</v-group>
</v-layer>`
})
// ----------------------------------------------------------------------------
// -------------------------------- Local -------------------------------------
// ----------------------------------------------------------------------------
Vue.component('rl-local', {
props: ['machine', 'maze', 'config'],
data: function () {
return {
robot_image: null,
energy_image: null,
}
},
watch: {
learning_rate: function(new_val) {
machine.lr = new_val;
render_latex();
created() {
set_images(this);
},
discount_factor: function(new_val) {
machine.df = new_val;
render_latex();
computed: {
main_config: function(){
return {
...this.config,
offset: {
x: -(this.config.width-this.base_size*3)/2,
y: -(this.config.height-this.base_size*3)/2,
}
}
},
epsilon: function(new_val) {
machine.epsilon = new_val;
map_config: function() {
return {
x: -(this.machine.state.x)*this.base_size,
y: -(this.machine.state.y)*this.base_size,
offset: {
x: -this.base_size,
y: -this.base_size,
}
}
})
function render_latex() {
// (1-lr) * Q[state, action] + lr * (reward + gamma * np.max(Q[new_state, :])
katex.render(`Q(s,a)\\leftarrow${(1-machine.lr).toFixed(2)}Q(s,a)+${machine.lr.toFixed(2)}(reward + ${machine.df.toFixed(2)}\\max_{a'}(Q(s_{new}, a'))`, document.getElementById('formula'),{displayMode: true,});
}
render_latex();
},
robot_config: function() {
return {
height: this.base_size,
width: this.base_size,
x: this.center,
y: this.center,
image: this.robot_image,
offset:{
x: this.base_size/2,
y: this.base_size/2,
}
}
},
energy_config: function() {
return {
height: this.base_size,
width: this.base_size,
offset: {
x: this.base_size/2,
y: this.base_size/2
},
image: this.energy_image,
}
},
base_size: function() {
return Math.min(this.config.height/3, this.config.width/3);
},
center: function() {
return 3*this.base_size / 2;
},
strokeW: function() {
return this.base_size / 50;
},
},
methods: {
get_tile_type: function(state) {
var pos = this.machine.s2p(state);
if (pos.y > maze.height) {
return null;
} else if (pos.x > maze.width) {
return null;
} else {
return maze.map[pos.y][pos.x];
}
},
in_plus: function(pos1, pos2) {
if (Math.abs(pos1.x - pos2.x) + Math.abs(pos1.y - pos2.y) < 2) {
return true;
}
return false;
},
get_field_config: function(state) {
var pos = this.machine.s2p(state);
return {
x: this.base_size * pos.x+this.base_size/2,
y: this.base_size * pos.y+this.base_size/2,
}
},
get_tile_config: function(i, t_type, local = true) {
// var pos = this.s2p(i);
var over = {};
// not in plus
if (local) {
if (!this.in_plus(this.machine.s2p(i), {
x: Math.round(this.machine.state.x),
y: Math.round(this.machine.state.y)
})) {
over = {
opacity: 0,
fill: "#eee"
};
} else if (i != this.machine.p2s(Math.round(this.machine.state.x), Math.round(this.machine.state.y))) {
over = {
opacity: 1,
fill: "#eee"
};
}
}
const layout = {
width: this.base_size,
height: this.base_size,
stroke: '#ddd',
strokeWidth: this.strokeW,
offset: {
x: this.base_size/2,
y: this.base_size/2,
}
};
switch (t_type) {
case tile.regular:
return {
...layout,
fill: '#fff',
opacity: 1,
...over,
}
case tile.end:
return {
...layout,
fill: '#0eb500',
opacity: 1,
...over,
}
case tile.start:
return {
...layout,
fill: '#ff0008',
opacity: 1,
...over,
}
case tile.dangerous:
return {
...layout,
fill: '#FF7B17',
opacity: 1,
...over,
}
case tile.wall:
return {
...layout,
opacity: 1,
...over,
fill: '#000000',
}
}
},
},
template:
`<v-layer ref="map_layer" :config="main_config">
<v-group ref="map_group" :config="map_config">
<v-group v-for="(t_type, idx) in maze.map.flat()" :config="get_field_config(idx)">
<v-rect :config="get_tile_config(idx, t_type)"></v-rect>
<v-image :config="energy_config" v-if="t_type==8 && idx==Math.round(machine.state.x)+maze.width*Math.round(machine.state.y)"></v-image>
</v-group>
</v-group>
<v-image :config="robot_config"></v-image>
</v-layer>`
})
// ----------------------------------------------------------------------------
// ------------------------------ lightbox ------------------------------------
// ----------------------------------------------------------------------------
Vue.component('light-box', {
props: ['content', 'options', 'active'],
// data: function () {
// return {
// active: false,
// }
// },
// methods: {
// close: function() {
// this.active = false;
// },
// open: function() {
// this.active = true;
// },
// },
template: `
<div class="lightbox" v-bind:class="{ active: active }">{{ content }}
<div class="options">
<button v-for="(item, key) in options" v-on:click="item">{{ key }}</button>
</div>
</div>`
})
// ----------------------------------------------------------------------------
// -------------------------------- Main --------------------------------------
// ----------------------------------------------------------------------------
function make_machine_reactive(th, machine){
var $this = th;
// Score wrapper
var s = machine.score;
$this.machine.score = s;
Object.defineProperty(machine, 'score', {
get: function() {
return this._score
},
set: function(ne) {
this._score = ne;
$this.machine.score = ne
}
});
machine.score = s;
// Score history wrapper
var s = machine.score_history;
$this.machine.score_history = s;
Object.defineProperty(machine, 'score_history', {
get: function() {
return this._score_history
},
set: function(ne) {
this._score_history = ne;
$this.machine.score_history = ne
}
});
machine.score_history = s;
// State wrapper
var s = machine.state;
$this.machine.state = $this.s2p(s);
Object.defineProperty(machine, 'state', {
get: function() {
return this._state
},
set: function(ne) {
this._state = ne;
$this.handleState(this._state);
}
});
machine.state = s;
$this.machine.s2p = $this.s2p;
$this.machine.p2s = $this.p2s;
window.addEventListener("episode", (ev) => {
if (ev.detail == "failed"){
$this.lightText = "Out of battery. The robot will be resetted.";
} else if (ev.detail == "success"){
$this.lightText = "You reached the goal. The robot will be resetted.";
}
$this.lightOptions = {
"ok": () => {$this.lightpopup = false;},
}
$this.lightpopup = true;
})
}
app = new Vue({
el: '#app',
components: {
VueSlider: window['vue-slider-component'],
},
data: {
state: 0,
maze: maze,
machine: {
object: machine,
q_table: machine.q_table,
state: {
x:0,
y:0,
},
state_tween: new TimelineLite(),
learning_rate: machine.lr,
discount_factor: machine.df,
epsilon: machine.epsilon,
score: machine.score,
score_history: machine.score_history,
s2p: null,
p2s: null,
},
width: 0,
height: 0,
state: null,
components: [],
lightText: "",
lightOptions: "",
lightpopup: false,
navigation: {},
onEnterState: function(){},
onLeaveState: function(){},
},
created() {
// Resize handler
window.addEventListener('resize', this.handleResize)
this.handleResize();
make_machine_reactive(this, machine);
this.state = "init";
},
destroyed() {
window.removeEventListener('resize', this.handleResize)
},
computed: {
stage_config: function() {
return {
width: this.width,
height: this.height,
}
},
map_config: function() {
return {
x: this.width*0.25,
y: this.height*0.1,
width: this.width*0.5,
height: this.height*0.8,
}
},
slider_config: function(){
return {
min: 0,
max: 1,
duration: 0,
interval: 0.01,
tooltip: 'none'
}
},
datacollection: function() {
return {
labels: Array.from(Array(this.machine.score_history.length).keys()),
datasets: [{
label: 'Data One',
backgroundColor: 'rgb(0,0,0,0)',
data: this.machine.score_history,//.simpleSMA(Math.round(50)),
fill: false,
borderColor: 'rgb(255, 159, 64)',
pointRadius: 1,
},
// {
// label: 'Data One',
// backgroundColor: 'rgb(0,0,0,0)',
// data: this.score_history.max(),
// fill: false,
// borderColor: 'rgb(64, 159, 255)',
// pointRadius: 1,
// },
]
}
},
plot_options: function() {
var $this = this;
return {
responsive: true,
maintainAspectRatio: false,
scales: {
xAxes: [{
// type: 'linear',
ticks: {
maxTicksLimit: 8,
maxRotation: 0,
}
}]
},
legend: {
display: false
}
}
},
},
methods: {
s2p: function(state) {
return {
x: (state % this.maze.width),
y: Math.floor(state / this.maze.width),
}
},
p2s: function(x, y) {
return x + y * this.maze.width;
},
handleState: function(s) {
if (!this.machine.object.running) {
this.machine.state_tween.to(this.machine.state, 0.2, {
x: this.s2p(s).x,
y: this.s2p(s).y
});
} else {
this.machine.state = this.s2p(s);
}
},
handleResize: function() {
this.width = window.innerWidth;
this.height = window.innerHeight;
},
isActive: function(what){
return this.components.indexOf(what) >= 0;
},
changeState: function(state){
this.components = [];
this.lightText = "";
this.lightOptions = "";
this.lightpopup = false;
this.navigation = {};
this.onEnterState = function(){};
this.onLeaveState = function(){};
this.state = state;
}
},
watch: {
'machine.learning_rate': function(new_val) {
machine.lr = new_val;
render_latex();
},
'machine.discount_factor': function(new_val) {
machine.df = new_val;
render_latex();
},
'machine.epsilon': function(new_val) {
machine.epsilon = new_val;
},
state: function(state){
this.onLeaveState();
Object.assign(this, StateMgr[state]);
this.onEnterState();
},
}
})
function render_latex() {
// (1-lr) * Q[state, action] + lr * (reward + gamma * np.max(Q[new_state, :])
katex.render(`Q(s,a)\\leftarrow${(1-machine.lr).toFixed(2)}Q(s,a)+${machine.lr.toFixed(2)}(reward + ${machine.df.toFixed(2)}\\max_{a'}(Q(s_{new}, a'))`, document.getElementById('formula'),{displayMode: true,});
}
render_latex();
// ----------------------------------------------------------------------------
// ------------------------------ StateMgr ------------------------------------
// ----------------------------------------------------------------------------
var StateMgr = {
init: {
lightText: `Reinforcement learning (RL) is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning. (wikipedia)
This exhibit explains how a robot can learn to navigate through a maze in order to reach its destination, before running out of power. At first the robot knows nothing, and learns from each new action (movement) and state (location reached). Slowly it starts to develop an understanding of the maze that will allow it to reach the charging station before it runs out of power. Eventually, it should learn to avoid any detour and reach the charging station in the optimal number of steps.`,
lightOptions: {
"next": () => app.changeState("local"),
},
onEnterState: function () {
this.lightpopup = true;
},
onLeaveState: function () {
this.lightpopup = false;
}
},
local: {
components: ["local", "navi", "score"],
navigation: {
"reset robot": () => machine.reset_machine(),
"continue": () => app.changeState("global"),
},
lightText: "But there is a problem! The robot cannot see the whole maze, it only knows where it is and in which direction it can move. Can you reach the charging station in those conditions? Use the arrows to move",
lightOptions: {
"next": () => {app.lightpopup = false;},
},
onEnterState: function () {
this.lightpopup = true;
},
onLeaveState: function () {
this.lightpopup = false;
}
},
global: {
components: ["global", "sliders", "plot", "navi", "score"],
navigation: {
"run 1 episode!": () => machine.run(1),
"run 100 episodes!": () => machine.run(100),
"auto step!": () => machine.auto_step(),
"greedy step!": () => machine.greedy_step(),
"reset machine": () => machine.reset_machine(),
},
lightText: `As a human, you keep track of where you are and how you got there without thinking, which helps you think about what actions you should take next to reach your destination. And you can also just look around! How can then the robot 'think' of the maze, to know which action is the best at every moment? And how can it learn that? It must somehow keep track of where it is, and remember how good or bad was each action at each place in the maze, try new things, and update it's "mental image" of what was a good decision and what not.
Reinforcement Learning uses the concept of a "Q-function", which keeps track of how "good" it expects it to be to take a specific action 'a' from a specific location 's'. This is written as Q(s, a). It also uses a "policy", which determines the best action to take in a given state, and is written as π(s). The robot must learn those functions while it navigates the maze. With each step, the functions are modified by a little bit, until eventually they give it the best strategy to solve the maze.`,
lightOptions: {
"continue": () => {app.lightpopup = false;},
},
onEnterState: function () {
this.lightpopup = true;
},
}
};
// ----------------------------------------------------------------------------
// ------------------------------- Dummy --------------------------------------
// ----------------------------------------------------------------------------
// Vue.component('dummy', {
// props: ['config'],
// data: function () {
// return {
// robot_image: null,
// }
// },
// created() {
//
// },
// mounted() {
//
// },
// computed: {
// fun: function() {
// return
// },
// },
// methods: {
// fun: function(arg) {
// },
// },
// template: ``
// })

Write Preview
Loading…
Cancel
Save