From 30c760c1e68b49e11a22ea0febe161e5eeb49530 Mon Sep 17 00:00:00 2001 From: Bernd Porr Date: Tue, 1 Jul 2014 22:14:42 +0100 Subject: [PATCH] staging: comedi: usbduxsigma: fixing rounding error in the range struct The AD converter on the usbsigma turns vref (2.5V/2) into the binary value 0x780000. However the full scale range goes up to 0x800000 so that vref/2 needs to be upscaled by 0x800000/0x780000 which is 1.06bar. In the driver it's been 1.06 without the infinite 6666. I've put the fraction in the code so that the compiler can calculate the most precise scaling value by itself. Signed-off-by: Bernd Porr Reviewed-by: Ian Abbott Signed-off-by: Greg Kroah-Hartman --- drivers/staging/comedi/drivers/usbduxsigma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/staging/comedi/drivers/usbduxsigma.c b/drivers/staging/comedi/drivers/usbduxsigma.c index ccc3ef7ba55c..e869af4784c6 100644 --- a/drivers/staging/comedi/drivers/usbduxsigma.c +++ b/drivers/staging/comedi/drivers/usbduxsigma.c @@ -136,7 +136,7 @@ static const struct comedi_lrange usbduxsigma_ai_range = { 1, { - BIP_RANGE(2.65 / 2.0) + BIP_RANGE(2.5 * 0x800000 / 0x780000 / 2.0) } }; -- 2.34.1