如何在画布上找到按钮的坐标，并在使用 Java 和 Selenium 后单击它们? [英] How to find the coordinates of the buttons on a canvas, and click on them after using Java and Selenium?

问题描述

我有一个在线计算器，我想自动执行一些操作，比如减法、除法等.但问题是没有元素，因为它是一个画布应用计算器(下面的链接).我想知道如何单击按钮来自动执行某些操作?

我正在尝试自动化的在线计算器:

I have an online calculator that i want to automate some operations for, like subtraction, division, etc. but the thing is that there are no elements since it is a canvas app calculator (link below). I'm wondering how can i click the buttons so im able to automate some operations?

The online calculator im trying to automate:

https://www.online-calculator.com/full-screen-calculator/

The canvas HTML code

<canvas id="canvas" width="335" height="434" style="position: absolute; display: block; background-color: rgb(255, 255, 255); width: 167.939px; height: 217px;"></canvas>

My Selenium-Java code

driver.get("https://www.online-calculator.com/full-screen-calculator/");
driver.manage().window().maximize();        
WebElement li = driver.findElements(By.tagName("iframe")).get(0);
driver.switchTo().frame(li);        
WebElement canv = driver.findElements(By.tagName("canvas")).get(0);
System.out.println(canv.getSize());
System.out.println(canv.getLocation());

try {
 Actions builder = new Actions(driver);
 builder.moveToElement(canv, x, y);
 builder.click();
 builder.perform();
    } catch (Exception e) 
    {
      // do nothing
    }

So as you see in above code, I still have to find the x and y of the operations i want to run. For ex, how can i find the coordinates of the buttons 10, 4 and the subtraction operation '-', if I want to calculate '10 - 4' and then validate that it is equal to 6. Any help would be highly appreciated. Thanks :)

Note: The canvas width, height and location will change if the window size changes ((Im thinking later of locking the window size so my tests are not flaky on different screens)).

解决方案

The <canvas> element is within an <iframe>. So to invoke click() on the elements within the <canvas> you have to:

• Induce WebDriverWait for the desired frame to be available and switch to it.
• Induce WebDriverWait for the desired element to be clickable.

• You can use the following solution:

  • Code Block:

    driver.get("https://www.online-calculator.com/full-screen-calculator/")
    new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe")));
    WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
    

---

HTML5 Canvas

The element is only a container for graphics and is a rectangular area on an HTML page. By default, a canvas has no border and no content. However, an id attribute (to be referred to in a script), a width and height attribute are specified to define the size of the canvas. To add a border, the style attribute is used. An example:

<canvas id="myCanvas" width="200" height="100" style="border:1px solid #000000;"></canvas>

The HTML canvas is a two-dimensional grid. The upper-left corner of the canvas has the coordinates (0,0).

In the article Automated Testing of HTML5 Canvas Applications with Selenium WebDriver @Aaron Mulder mentions, interacting with the elements within <canvas> is possible using event support of the Actions Class API:

• moveToElement(WebElement target, int xOffset, int yOffset): Moves the mouse to an offset from the top-left corner of the element.

  new Actions(driver).moveToElement(canvas, xWithinCanvas, yWithinCanvas).click().perform();
  

But is not 100% reliable as, every mouse down, mouse up, or mouse click happens at the center of the element. So the code above produces a mouse move event to the provided coordinates (x,y), then a mouse move event to the center of the <canvas>, then a mouse down, mouse up, and click at the center of the <canvas>. That should have been fine for a <button> but is not worth for a <canvas>, where you want to be able to click at a specific location.

The workaround, is to dispatch synthesized mouse events using JavaScript as follows:

// pageX and pageY are offsets which you need to know through mouse coordinates.
((JavascriptExecutor)driver).executeScript("var evt = $.Event('click', { pageX: " + x +
   ", pageY: " + (y + 55) + " } );" +
   "$('#myCanvas').trigger(evt);");

However, to click on the elements within the <canvas> you can be at ease using firefox as the mouse move event works well in Firefox and you can avoid using the mouse coordinates as the event processing as follows:

new Actions(driver).moveToElement(
   canvas, xWithinCanvas, yWithinCanvas).perform();
((JavascriptExecutor)driver).executeScript("$('#canvas').click();");

---

This usecase

To automate a substruction operation e.g. 3-1= using Selenium you can use the following solution:

• Code Block:

  driver.get("https://www.online-calculator.com/full-screen-calculator/");
  new WebDriverWait(driver, 10).until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.id("fullframe")));
  WebElement canvas = new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("canvas")));
  //clicking on 3
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(0,(255/6)*3).click().build().perform();
  //clicking on the substract sign (-)
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*2,(255/6)*3).click().build().perform();
  //clicking on 1
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset(-(174/5)*4,(255/6)*3).click().build().perform();
  //clicking on equals to sign (=)
  new Actions(driver).moveToElement(canvas, 0, 0).moveByOffset((174/5)*4,(255/6)*4).click().build().perform();
  

• Execution Video:

---

Reference

You can find a couple of relevant detailed discussion in:

• How to perform Mouse Wheel scrolling over HTML5 Canvas in Selenium?

这篇关于如何在画布上找到按钮的坐标，并在使用 Java 和 Selenium 后单击它们?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！